2023-02-11 14:59:59,203 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab 2023-02-11 14:59:59,216 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-11 14:59:59,231 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-11 14:59:59,231 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f, deleteOnExit=true 2023-02-11 14:59:59,231 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-11 14:59:59,232 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/test.cache.data in system properties and HBase conf 2023-02-11 14:59:59,232 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/hadoop.tmp.dir in system properties and HBase conf 2023-02-11 14:59:59,232 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/hadoop.log.dir in system properties and HBase conf 2023-02-11 14:59:59,233 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-11 14:59:59,233 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-11 14:59:59,233 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-11 14:59:59,352 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-11 14:59:59,722 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-11 14:59:59,726 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-11 14:59:59,726 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-11 14:59:59,726 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-11 14:59:59,727 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 14:59:59,727 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-11 14:59:59,727 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-11 14:59:59,727 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 14:59:59,728 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 14:59:59,728 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-11 14:59:59,728 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/nfs.dump.dir in system properties and HBase conf 2023-02-11 14:59:59,729 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/java.io.tmpdir in system properties and HBase conf 2023-02-11 14:59:59,729 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 14:59:59,729 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-11 14:59:59,729 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-11 15:00:00,219 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 15:00:00,223 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 15:00:00,499 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-11 15:00:00,650 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-11 15:00:00,665 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 15:00:00,701 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 15:00:00,733 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/java.io.tmpdir/Jetty_localhost_43555_hdfs____ks49r4/webapp 2023-02-11 15:00:00,863 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43555 2023-02-11 15:00:00,902 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 15:00:00,902 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 15:00:01,288 WARN [Listener at localhost/37623] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 15:00:01,383 WARN [Listener at localhost/37623] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 15:00:01,402 WARN [Listener at localhost/37623] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 15:00:01,408 INFO [Listener at localhost/37623] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 15:00:01,413 INFO [Listener at localhost/37623] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/java.io.tmpdir/Jetty_localhost_36643_datanode____.tc9dev/webapp 2023-02-11 15:00:01,516 INFO [Listener at localhost/37623] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36643 2023-02-11 15:00:01,819 WARN [Listener at localhost/45049] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 15:00:01,837 WARN [Listener at localhost/45049] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 15:00:01,841 WARN [Listener at localhost/45049] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 15:00:01,843 INFO [Listener at localhost/45049] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 15:00:01,851 INFO [Listener at localhost/45049] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/java.io.tmpdir/Jetty_localhost_43375_datanode____.k6kr6v/webapp 2023-02-11 15:00:01,987 INFO [Listener at localhost/45049] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43375 2023-02-11 15:00:01,997 WARN [Listener at localhost/34369] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 15:00:02,019 WARN [Listener at localhost/34369] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 15:00:02,021 WARN [Listener at localhost/34369] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 15:00:02,023 INFO [Listener at localhost/34369] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 15:00:02,027 INFO [Listener at localhost/34369] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/java.io.tmpdir/Jetty_localhost_35645_datanode____.ulo550/webapp 2023-02-11 15:00:02,148 INFO [Listener at localhost/34369] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35645 2023-02-11 15:00:02,156 WARN [Listener at localhost/39605] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 15:00:02,331 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1b410a30e6cd421d: Processing first storage report for DS-a33290da-ad13-4ce6-ba63-edb151184900 from datanode bf0add4a-0a9f-4550-8ba1-632114d3a2f1 2023-02-11 15:00:02,334 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1b410a30e6cd421d: from storage DS-a33290da-ad13-4ce6-ba63-edb151184900 node DatanodeRegistration(127.0.0.1:45987, datanodeUuid=bf0add4a-0a9f-4550-8ba1-632114d3a2f1, infoPort=41747, infoSecurePort=0, ipcPort=39605, storageInfo=lv=-57;cid=testClusterID;nsid=1367286111;c=1676127600293), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-11 15:00:02,334 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x52cd7a835814d456: Processing first storage report for DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0 from datanode 2ed47ade-61b2-412c-b2fa-38a8167f2f53 2023-02-11 15:00:02,334 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x52cd7a835814d456: from storage DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0 node DatanodeRegistration(127.0.0.1:38149, datanodeUuid=2ed47ade-61b2-412c-b2fa-38a8167f2f53, infoPort=35073, infoSecurePort=0, ipcPort=34369, storageInfo=lv=-57;cid=testClusterID;nsid=1367286111;c=1676127600293), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 15:00:02,334 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4288ffd755ba21ef: Processing first storage report for DS-27c45a68-a2cd-45c8-941a-921f2704e91e from datanode ffa7b8bb-5ab6-498c-b50f-1688dc4647c1 2023-02-11 15:00:02,334 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4288ffd755ba21ef: from storage DS-27c45a68-a2cd-45c8-941a-921f2704e91e node DatanodeRegistration(127.0.0.1:34407, datanodeUuid=ffa7b8bb-5ab6-498c-b50f-1688dc4647c1, infoPort=46117, infoSecurePort=0, ipcPort=45049, storageInfo=lv=-57;cid=testClusterID;nsid=1367286111;c=1676127600293), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 15:00:02,335 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1b410a30e6cd421d: Processing first storage report for DS-b754bb55-222c-4c4a-9d25-fbe572b8dacf from datanode bf0add4a-0a9f-4550-8ba1-632114d3a2f1 2023-02-11 15:00:02,335 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1b410a30e6cd421d: from storage DS-b754bb55-222c-4c4a-9d25-fbe572b8dacf node DatanodeRegistration(127.0.0.1:45987, datanodeUuid=bf0add4a-0a9f-4550-8ba1-632114d3a2f1, infoPort=41747, infoSecurePort=0, ipcPort=39605, storageInfo=lv=-57;cid=testClusterID;nsid=1367286111;c=1676127600293), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 15:00:02,335 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x52cd7a835814d456: Processing first storage report for DS-0f3285a9-5e69-4228-8c3d-d5698b645a37 from datanode 2ed47ade-61b2-412c-b2fa-38a8167f2f53 2023-02-11 15:00:02,335 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x52cd7a835814d456: from storage DS-0f3285a9-5e69-4228-8c3d-d5698b645a37 node DatanodeRegistration(127.0.0.1:38149, datanodeUuid=2ed47ade-61b2-412c-b2fa-38a8167f2f53, infoPort=35073, infoSecurePort=0, ipcPort=34369, storageInfo=lv=-57;cid=testClusterID;nsid=1367286111;c=1676127600293), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 15:00:02,335 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4288ffd755ba21ef: Processing first storage report for DS-af683fc9-995b-49b4-bc47-963486478c70 from datanode ffa7b8bb-5ab6-498c-b50f-1688dc4647c1 2023-02-11 15:00:02,336 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4288ffd755ba21ef: from storage DS-af683fc9-995b-49b4-bc47-963486478c70 node DatanodeRegistration(127.0.0.1:34407, datanodeUuid=ffa7b8bb-5ab6-498c-b50f-1688dc4647c1, infoPort=46117, infoSecurePort=0, ipcPort=45049, storageInfo=lv=-57;cid=testClusterID;nsid=1367286111;c=1676127600293), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-11 15:00:02,570 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab 2023-02-11 15:00:02,631 INFO [Listener at localhost/39605] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/zookeeper_0, clientPort=58940, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-11 15:00:02,644 INFO [Listener at localhost/39605] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58940 2023-02-11 15:00:02,656 INFO [Listener at localhost/39605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:02,659 INFO [Listener at localhost/39605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:03,345 INFO [Listener at localhost/39605] util.FSUtils(479): Created version file at hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92 with version=8 2023-02-11 15:00:03,346 INFO [Listener at localhost/39605] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/hbase-staging 2023-02-11 15:00:03,647 INFO [Listener at localhost/39605] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-11 15:00:04,113 INFO [Listener at localhost/39605] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 15:00:04,144 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,145 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,146 INFO [Listener at localhost/39605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 15:00:04,146 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,146 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 15:00:04,303 INFO [Listener at localhost/39605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 15:00:04,379 DEBUG [Listener at localhost/39605] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-11 15:00:04,474 INFO [Listener at localhost/39605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38517 2023-02-11 15:00:04,485 INFO [Listener at localhost/39605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:04,487 INFO [Listener at localhost/39605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:04,507 INFO [Listener at localhost/39605] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38517 connecting to ZooKeeper ensemble=127.0.0.1:58940 2023-02-11 15:00:04,547 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:385170x0, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 15:00:04,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:38517-0x10043e5717e0000 connected 2023-02-11 15:00:04,574 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 15:00:04,575 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 15:00:04,578 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 15:00:04,587 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38517 2023-02-11 15:00:04,587 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38517 2023-02-11 15:00:04,588 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38517 2023-02-11 15:00:04,592 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38517 2023-02-11 15:00:04,593 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38517 2023-02-11 15:00:04,601 INFO [Listener at localhost/39605] master.HMaster(439): hbase.rootdir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92, hbase.cluster.distributed=false 2023-02-11 15:00:04,676 INFO [Listener at localhost/39605] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 15:00:04,676 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,676 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,677 INFO [Listener at localhost/39605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 15:00:04,677 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,677 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 15:00:04,682 INFO [Listener at localhost/39605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 15:00:04,686 INFO [Listener at localhost/39605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41207 2023-02-11 15:00:04,689 INFO [Listener at localhost/39605] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 15:00:04,697 INFO [Listener at localhost/39605] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 15:00:04,698 INFO [Listener at localhost/39605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:04,701 INFO [Listener at localhost/39605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:04,703 INFO [Listener at localhost/39605] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41207 connecting to ZooKeeper ensemble=127.0.0.1:58940 2023-02-11 15:00:04,711 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:412070x0, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 15:00:04,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41207-0x10043e5717e0001 connected 2023-02-11 15:00:04,712 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 15:00:04,713 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 15:00:04,715 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 15:00:04,720 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41207 2023-02-11 15:00:04,720 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41207 2023-02-11 15:00:04,720 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41207 2023-02-11 15:00:04,723 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41207 2023-02-11 15:00:04,724 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41207 2023-02-11 15:00:04,743 INFO [Listener at localhost/39605] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 15:00:04,744 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,744 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,745 INFO [Listener at localhost/39605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 15:00:04,745 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,745 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 15:00:04,745 INFO [Listener at localhost/39605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 15:00:04,747 INFO [Listener at localhost/39605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33165 2023-02-11 15:00:04,748 INFO [Listener at localhost/39605] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 15:00:04,749 INFO [Listener at localhost/39605] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 15:00:04,750 INFO [Listener at localhost/39605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:04,753 INFO [Listener at localhost/39605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:04,756 INFO [Listener at localhost/39605] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33165 connecting to ZooKeeper ensemble=127.0.0.1:58940 2023-02-11 15:00:04,760 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:331650x0, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 15:00:04,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33165-0x10043e5717e0002 connected 2023-02-11 15:00:04,762 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 15:00:04,763 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 15:00:04,764 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 15:00:04,764 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33165 2023-02-11 15:00:04,765 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33165 2023-02-11 15:00:04,765 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33165 2023-02-11 15:00:04,766 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33165 2023-02-11 15:00:04,766 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33165 2023-02-11 15:00:04,786 INFO [Listener at localhost/39605] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 15:00:04,786 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,786 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,786 INFO [Listener at localhost/39605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 15:00:04,786 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 15:00:04,787 INFO [Listener at localhost/39605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 15:00:04,787 INFO [Listener at localhost/39605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 15:00:04,788 INFO [Listener at localhost/39605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42505 2023-02-11 15:00:04,789 INFO [Listener at localhost/39605] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 15:00:04,790 INFO [Listener at localhost/39605] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 15:00:04,791 INFO [Listener at localhost/39605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:04,793 INFO [Listener at localhost/39605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:04,795 INFO [Listener at localhost/39605] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42505 connecting to ZooKeeper ensemble=127.0.0.1:58940 2023-02-11 15:00:04,800 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:425050x0, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 15:00:04,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42505-0x10043e5717e0003 connected 2023-02-11 15:00:04,801 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 15:00:04,802 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 15:00:04,803 DEBUG [Listener at localhost/39605] zookeeper.ZKUtil(164): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 15:00:04,803 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42505 2023-02-11 15:00:04,804 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42505 2023-02-11 15:00:04,804 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42505 2023-02-11 15:00:04,804 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42505 2023-02-11 15:00:04,805 DEBUG [Listener at localhost/39605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42505 2023-02-11 15:00:04,806 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,38517,1676127603486 2023-02-11 15:00:04,815 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 15:00:04,817 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,38517,1676127603486 2023-02-11 15:00:04,836 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 15:00:04,836 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 15:00:04,836 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 15:00:04,836 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 15:00:04,836 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 15:00:04,838 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 15:00:04,840 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,38517,1676127603486 from backup master directory 2023-02-11 15:00:04,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 15:00:04,844 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,38517,1676127603486 2023-02-11 15:00:04,844 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 15:00:04,845 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 15:00:04,845 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,38517,1676127603486 2023-02-11 15:00:04,847 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-11 15:00:04,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-11 15:00:04,944 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/hbase.id with ID: 5c3dc46b-6e4d-4736-8215-1b06d9b3268d 2023-02-11 15:00:04,990 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 15:00:05,008 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 15:00:05,055 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7ef34e61 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:05,088 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e68db18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:05,113 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-11 15:00:05,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-11 15:00:05,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-11 15:00:05,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-11 15:00:05,134 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 15:00:05,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 15:00:05,140 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 15:00:05,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/data/master/store-tmp 2023-02-11 15:00:05,215 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:05,216 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 15:00:05,216 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 15:00:05,216 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 15:00:05,216 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 15:00:05,216 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 15:00:05,216 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 15:00:05,216 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676127605215Disabling compacts and flushes for region at 1676127605216 (+1 ms)Disabling writes for close at 1676127605216Writing region close event to WAL at 1676127605216Closed at 1676127605216 2023-02-11 15:00:05,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/WALs/jenkins-hbase4.apache.org,38517,1676127603486 2023-02-11 15:00:05,238 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38517%2C1676127603486, suffix=, logDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/WALs/jenkins-hbase4.apache.org,38517,1676127603486, archiveDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/oldWALs, maxLogs=10 2023-02-11 15:00:05,294 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45987,DS-a33290da-ad13-4ce6-ba63-edb151184900,DISK] 2023-02-11 15:00:05,294 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38149,DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0,DISK] 2023-02-11 15:00:05,294 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34407,DS-27c45a68-a2cd-45c8-941a-921f2704e91e,DISK] 2023-02-11 15:00:05,302 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-11 15:00:05,370 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/WALs/jenkins-hbase4.apache.org,38517,1676127603486/jenkins-hbase4.apache.org%2C38517%2C1676127603486.1676127605248 2023-02-11 15:00:05,370 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38149,DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0,DISK], DatanodeInfoWithStorage[127.0.0.1:34407,DS-27c45a68-a2cd-45c8-941a-921f2704e91e,DISK], DatanodeInfoWithStorage[127.0.0.1:45987,DS-a33290da-ad13-4ce6-ba63-edb151184900,DISK]] 2023-02-11 15:00:05,371 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-11 15:00:05,371 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:05,375 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 15:00:05,376 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 15:00:05,434 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-11 15:00:05,440 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-11 15:00:05,463 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-11 15:00:05,475 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:05,482 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 15:00:05,484 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 15:00:05,502 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 15:00:05,507 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:05,508 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67866691, jitterRate=0.011292502284049988}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:05,508 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676127605402Initializing all the Stores at 1676127605404 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676127605405 (+1 ms)Cleaning up temporary data from old regions at 1676127605492 (+87 ms)Cleaning up detritus from prior splits at 1676127605493 (+1 ms)Region opened successfully at 1676127605508 (+15 ms) 2023-02-11 15:00:05,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-11 15:00:05,530 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-11 15:00:05,530 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-11 15:00:05,532 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-11 15:00:05,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-11 15:00:05,565 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-11 15:00:05,566 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-11 15:00:05,590 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-11 15:00:05,595 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-11 15:00:05,625 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-11 15:00:05,631 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-11 15:00:05,632 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-11 15:00:05,638 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-11 15:00:05,642 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-11 15:00:05,646 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 15:00:05,648 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-11 15:00:05,648 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-11 15:00:05,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-11 15:00:05,664 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 15:00:05,664 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 15:00:05,664 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 15:00:05,664 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 15:00:05,665 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 15:00:05,665 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,38517,1676127603486, sessionid=0x10043e5717e0000, setting cluster-up flag (Was=false) 2023-02-11 15:00:05,683 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 15:00:05,690 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-11 15:00:05,692 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38517,1676127603486 2023-02-11 15:00:05,698 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 15:00:05,704 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-11 15:00:05,706 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38517,1676127603486 2023-02-11 15:00:05,708 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp 2023-02-11 15:00:05,809 INFO [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(952): ClusterId : 5c3dc46b-6e4d-4736-8215-1b06d9b3268d 2023-02-11 15:00:05,810 INFO [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(952): ClusterId : 5c3dc46b-6e4d-4736-8215-1b06d9b3268d 2023-02-11 15:00:05,809 INFO [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(952): ClusterId : 5c3dc46b-6e4d-4736-8215-1b06d9b3268d 2023-02-11 15:00:05,816 DEBUG [RS:1;jenkins-hbase4:33165] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 15:00:05,816 DEBUG [RS:0;jenkins-hbase4:41207] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 15:00:05,816 DEBUG [RS:2;jenkins-hbase4:42505] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 15:00:05,824 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-11 15:00:05,825 DEBUG [RS:1;jenkins-hbase4:33165] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 15:00:05,825 DEBUG [RS:0;jenkins-hbase4:41207] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 15:00:05,825 DEBUG [RS:2;jenkins-hbase4:42505] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 15:00:05,825 DEBUG [RS:0;jenkins-hbase4:41207] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 15:00:05,825 DEBUG [RS:1;jenkins-hbase4:33165] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 15:00:05,825 DEBUG [RS:2;jenkins-hbase4:42505] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 15:00:05,829 DEBUG [RS:0;jenkins-hbase4:41207] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 15:00:05,829 DEBUG [RS:2;jenkins-hbase4:42505] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 15:00:05,829 DEBUG [RS:1;jenkins-hbase4:33165] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 15:00:05,833 DEBUG [RS:2;jenkins-hbase4:42505] zookeeper.ReadOnlyZKClient(139): Connect 0x65397fd3 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:05,834 DEBUG [RS:1;jenkins-hbase4:33165] zookeeper.ReadOnlyZKClient(139): Connect 0x14a0e377 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:05,834 DEBUG [RS:0;jenkins-hbase4:41207] zookeeper.ReadOnlyZKClient(139): Connect 0x62f1e446 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:05,858 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 15:00:05,859 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 15:00:05,859 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 15:00:05,859 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 15:00:05,859 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-11 15:00:05,859 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:05,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 15:00:05,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:05,863 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676127635863 2023-02-11 15:00:05,864 DEBUG [RS:0;jenkins-hbase4:41207] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c360668, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:05,864 DEBUG [RS:1;jenkins-hbase4:33165] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a670f93, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:05,864 DEBUG [RS:2;jenkins-hbase4:42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bfe9d04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:05,865 DEBUG [RS:0;jenkins-hbase4:41207] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68098f36, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 15:00:05,865 DEBUG [RS:1;jenkins-hbase4:33165] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e588807, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 15:00:05,865 DEBUG [RS:2;jenkins-hbase4:42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fc114a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 15:00:05,866 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-11 15:00:05,874 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 15:00:05,874 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-11 15:00:05,879 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-11 15:00:05,880 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 15:00:05,887 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-11 15:00:05,888 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-11 15:00:05,888 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-11 15:00:05,889 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-11 15:00:05,894 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:05,895 DEBUG [RS:1;jenkins-hbase4:33165] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:33165 2023-02-11 15:00:05,898 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-11 15:00:05,898 DEBUG [RS:2;jenkins-hbase4:42505] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:42505 2023-02-11 15:00:05,901 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-11 15:00:05,901 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-11 15:00:05,902 DEBUG [RS:0;jenkins-hbase4:41207] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:41207 2023-02-11 15:00:05,903 INFO [RS:1;jenkins-hbase4:33165] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 15:00:05,903 INFO [RS:0;jenkins-hbase4:41207] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 15:00:05,904 INFO [RS:0;jenkins-hbase4:41207] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 15:00:05,903 INFO [RS:2;jenkins-hbase4:42505] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 15:00:05,904 DEBUG [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 15:00:05,904 INFO [RS:1;jenkins-hbase4:33165] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 15:00:05,904 INFO [RS:2;jenkins-hbase4:42505] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 15:00:05,904 DEBUG [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 15:00:05,904 DEBUG [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 15:00:05,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-11 15:00:05,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-11 15:00:05,908 INFO [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38517,1676127603486 with isa=jenkins-hbase4.apache.org/172.31.14.131:41207, startcode=1676127604675 2023-02-11 15:00:05,908 INFO [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38517,1676127603486 with isa=jenkins-hbase4.apache.org/172.31.14.131:33165, startcode=1676127604743 2023-02-11 15:00:05,908 INFO [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38517,1676127603486 with isa=jenkins-hbase4.apache.org/172.31.14.131:42505, startcode=1676127604785 2023-02-11 15:00:05,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676127605909,5,FailOnTimeoutGroup] 2023-02-11 15:00:05,911 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676127605911,5,FailOnTimeoutGroup] 2023-02-11 15:00:05,912 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:05,912 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-11 15:00:05,914 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:05,914 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:05,936 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 15:00:05,936 DEBUG [RS:1;jenkins-hbase4:33165] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 15:00:05,936 DEBUG [RS:0;jenkins-hbase4:41207] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 15:00:05,936 DEBUG [RS:2;jenkins-hbase4:42505] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 15:00:05,937 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 15:00:05,937 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92 2023-02-11 15:00:05,966 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:05,970 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 15:00:05,975 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/info 2023-02-11 15:00:05,976 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 15:00:05,978 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:05,979 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 15:00:05,982 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/rep_barrier 2023-02-11 15:00:05,983 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 15:00:05,984 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:05,984 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 15:00:05,986 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/table 2023-02-11 15:00:05,987 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 15:00:05,988 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:05,990 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740 2023-02-11 15:00:05,991 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740 2023-02-11 15:00:05,995 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 15:00:05,998 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 15:00:06,002 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:06,003 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62002817, jitterRate=-0.07608602941036224}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 15:00:06,003 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676127605967Initializing all the Stores at 1676127605970 (+3 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676127605970Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676127605970Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676127605970Cleaning up temporary data from old regions at 1676127605993 (+23 ms)Cleaning up detritus from prior splits at 1676127605993Region opened successfully at 1676127606003 (+10 ms) 2023-02-11 15:00:06,003 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 15:00:06,003 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 15:00:06,003 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 15:00:06,003 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 15:00:06,003 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 15:00:06,008 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 15:00:06,008 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676127606003Disabling compacts and flushes for region at 1676127606003Disabling writes for close at 1676127606003Writing region close event to WAL at 1676127606007 (+4 ms)Closed at 1676127606008 (+1 ms) 2023-02-11 15:00:06,009 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35511, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 15:00:06,009 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49173, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 15:00:06,009 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49215, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 15:00:06,016 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 15:00:06,016 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-11 15:00:06,024 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38517] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:06,024 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38517] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:06,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:06,029 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-11 15:00:06,044 DEBUG [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92 2023-02-11 15:00:06,044 DEBUG [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92 2023-02-11 15:00:06,044 DEBUG [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37623 2023-02-11 15:00:06,044 DEBUG [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92 2023-02-11 15:00:06,044 DEBUG [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 15:00:06,044 DEBUG [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37623 2023-02-11 15:00:06,044 DEBUG [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37623 2023-02-11 15:00:06,045 DEBUG [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 15:00:06,044 DEBUG [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 15:00:06,047 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-11 15:00:06,053 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:06,054 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 15:00:06,054 DEBUG [RS:0;jenkins-hbase4:41207] zookeeper.ZKUtil(162): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:06,055 WARN [RS:0;jenkins-hbase4:41207] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 15:00:06,056 INFO [RS:0;jenkins-hbase4:41207] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 15:00:06,056 DEBUG [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:06,056 DEBUG [RS:2;jenkins-hbase4:42505] zookeeper.ZKUtil(162): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:06,056 WARN [RS:2;jenkins-hbase4:42505] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 15:00:06,056 DEBUG [RS:1;jenkins-hbase4:33165] zookeeper.ZKUtil(162): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:06,057 INFO [RS:2;jenkins-hbase4:42505] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 15:00:06,057 WARN [RS:1;jenkins-hbase4:33165] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 15:00:06,057 DEBUG [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:06,058 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33165,1676127604743] 2023-02-11 15:00:06,058 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42505,1676127604785] 2023-02-11 15:00:06,058 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41207,1676127604675] 2023-02-11 15:00:06,058 INFO [RS:1;jenkins-hbase4:33165] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 15:00:06,058 DEBUG [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:06,073 DEBUG [RS:2;jenkins-hbase4:42505] zookeeper.ZKUtil(162): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:06,073 DEBUG [RS:1;jenkins-hbase4:33165] zookeeper.ZKUtil(162): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:06,073 DEBUG [RS:0;jenkins-hbase4:41207] zookeeper.ZKUtil(162): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:06,073 DEBUG [RS:2;jenkins-hbase4:42505] zookeeper.ZKUtil(162): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:06,074 DEBUG [RS:0;jenkins-hbase4:41207] zookeeper.ZKUtil(162): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:06,074 DEBUG [RS:1;jenkins-hbase4:33165] zookeeper.ZKUtil(162): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:06,074 DEBUG [RS:2;jenkins-hbase4:42505] zookeeper.ZKUtil(162): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:06,074 DEBUG [RS:0;jenkins-hbase4:41207] zookeeper.ZKUtil(162): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:06,074 DEBUG [RS:1;jenkins-hbase4:33165] zookeeper.ZKUtil(162): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:06,084 DEBUG [RS:0;jenkins-hbase4:41207] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 15:00:06,084 DEBUG [RS:2;jenkins-hbase4:42505] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 15:00:06,084 DEBUG [RS:1;jenkins-hbase4:33165] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 15:00:06,094 INFO [RS:1;jenkins-hbase4:33165] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 15:00:06,094 INFO [RS:0;jenkins-hbase4:41207] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 15:00:06,094 INFO [RS:2;jenkins-hbase4:42505] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 15:00:06,114 INFO [RS:0;jenkins-hbase4:41207] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 15:00:06,114 INFO [RS:2;jenkins-hbase4:42505] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 15:00:06,114 INFO [RS:1;jenkins-hbase4:33165] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 15:00:06,118 INFO [RS:2;jenkins-hbase4:42505] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 15:00:06,118 INFO [RS:1;jenkins-hbase4:33165] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 15:00:06,119 INFO [RS:2;jenkins-hbase4:42505] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,119 INFO [RS:1;jenkins-hbase4:33165] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,118 INFO [RS:0;jenkins-hbase4:41207] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 15:00:06,119 INFO [RS:0;jenkins-hbase4:41207] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,120 INFO [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 15:00:06,120 INFO [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 15:00:06,120 INFO [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 15:00:06,128 INFO [RS:2;jenkins-hbase4:42505] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,128 INFO [RS:0;jenkins-hbase4:41207] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,128 INFO [RS:1;jenkins-hbase4:33165] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,128 DEBUG [RS:2;jenkins-hbase4:42505] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,128 DEBUG [RS:0;jenkins-hbase4:41207] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:2;jenkins-hbase4:42505] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:0;jenkins-hbase4:41207] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:2;jenkins-hbase4:42505] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:0;jenkins-hbase4:41207] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:2;jenkins-hbase4:42505] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:0;jenkins-hbase4:41207] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:2;jenkins-hbase4:42505] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:0;jenkins-hbase4:41207] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:2;jenkins-hbase4:42505] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 15:00:06,129 DEBUG [RS:0;jenkins-hbase4:41207] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 15:00:06,129 DEBUG [RS:2;jenkins-hbase4:42505] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:0;jenkins-hbase4:41207] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,129 DEBUG [RS:1;jenkins-hbase4:33165] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,130 DEBUG [RS:0;jenkins-hbase4:41207] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,130 DEBUG [RS:1;jenkins-hbase4:33165] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,130 DEBUG [RS:0;jenkins-hbase4:41207] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,130 DEBUG [RS:1;jenkins-hbase4:33165] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,130 DEBUG [RS:0;jenkins-hbase4:41207] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,130 DEBUG [RS:1;jenkins-hbase4:33165] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,130 DEBUG [RS:2;jenkins-hbase4:42505] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,130 DEBUG [RS:1;jenkins-hbase4:33165] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,130 DEBUG [RS:2;jenkins-hbase4:42505] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,131 DEBUG [RS:1;jenkins-hbase4:33165] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 15:00:06,131 DEBUG [RS:2;jenkins-hbase4:42505] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,131 DEBUG [RS:1;jenkins-hbase4:33165] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,131 DEBUG [RS:1;jenkins-hbase4:33165] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,131 DEBUG [RS:1;jenkins-hbase4:33165] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,131 DEBUG [RS:1;jenkins-hbase4:33165] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 15:00:06,132 INFO [RS:0;jenkins-hbase4:41207] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,132 INFO [RS:0;jenkins-hbase4:41207] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,133 INFO [RS:0;jenkins-hbase4:41207] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,133 INFO [RS:1;jenkins-hbase4:33165] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,133 INFO [RS:2;jenkins-hbase4:42505] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,133 INFO [RS:1;jenkins-hbase4:33165] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,133 INFO [RS:2;jenkins-hbase4:42505] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,133 INFO [RS:1;jenkins-hbase4:33165] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,134 INFO [RS:2;jenkins-hbase4:42505] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,152 INFO [RS:0;jenkins-hbase4:41207] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 15:00:06,152 INFO [RS:2;jenkins-hbase4:42505] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 15:00:06,152 INFO [RS:1;jenkins-hbase4:33165] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 15:00:06,155 INFO [RS:2;jenkins-hbase4:42505] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42505,1676127604785-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,155 INFO [RS:1;jenkins-hbase4:33165] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33165,1676127604743-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,155 INFO [RS:0;jenkins-hbase4:41207] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41207,1676127604675-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,179 INFO [RS:2;jenkins-hbase4:42505] regionserver.Replication(203): jenkins-hbase4.apache.org,42505,1676127604785 started 2023-02-11 15:00:06,179 INFO [RS:0;jenkins-hbase4:41207] regionserver.Replication(203): jenkins-hbase4.apache.org,41207,1676127604675 started 2023-02-11 15:00:06,179 INFO [RS:1;jenkins-hbase4:33165] regionserver.Replication(203): jenkins-hbase4.apache.org,33165,1676127604743 started 2023-02-11 15:00:06,179 INFO [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,41207,1676127604675, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41207, sessionid=0x10043e5717e0001 2023-02-11 15:00:06,179 INFO [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42505,1676127604785, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42505, sessionid=0x10043e5717e0003 2023-02-11 15:00:06,180 INFO [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33165,1676127604743, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33165, sessionid=0x10043e5717e0002 2023-02-11 15:00:06,180 DEBUG [RS:2;jenkins-hbase4:42505] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 15:00:06,180 DEBUG [RS:1;jenkins-hbase4:33165] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 15:00:06,180 DEBUG [RS:0;jenkins-hbase4:41207] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 15:00:06,181 DEBUG [RS:0;jenkins-hbase4:41207] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:06,180 DEBUG [RS:1;jenkins-hbase4:33165] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:06,180 DEBUG [RS:2;jenkins-hbase4:42505] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:06,181 DEBUG [RS:1;jenkins-hbase4:33165] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33165,1676127604743' 2023-02-11 15:00:06,181 DEBUG [RS:0;jenkins-hbase4:41207] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41207,1676127604675' 2023-02-11 15:00:06,181 DEBUG [RS:1;jenkins-hbase4:33165] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 15:00:06,181 DEBUG [RS:2;jenkins-hbase4:42505] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42505,1676127604785' 2023-02-11 15:00:06,181 DEBUG [RS:0;jenkins-hbase4:41207] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 15:00:06,182 DEBUG [RS:2;jenkins-hbase4:42505] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 15:00:06,182 DEBUG [RS:1;jenkins-hbase4:33165] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 15:00:06,182 DEBUG [RS:2;jenkins-hbase4:42505] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 15:00:06,182 DEBUG [RS:0;jenkins-hbase4:41207] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 15:00:06,183 DEBUG [RS:1;jenkins-hbase4:33165] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 15:00:06,183 DEBUG [RS:0;jenkins-hbase4:41207] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 15:00:06,183 DEBUG [RS:1;jenkins-hbase4:33165] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 15:00:06,183 DEBUG [RS:2;jenkins-hbase4:42505] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 15:00:06,183 DEBUG [RS:1;jenkins-hbase4:33165] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:06,183 DEBUG [RS:0;jenkins-hbase4:41207] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 15:00:06,183 DEBUG [RS:1;jenkins-hbase4:33165] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33165,1676127604743' 2023-02-11 15:00:06,183 DEBUG [RS:2;jenkins-hbase4:42505] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 15:00:06,184 DEBUG [RS:1;jenkins-hbase4:33165] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 15:00:06,183 DEBUG [RS:0;jenkins-hbase4:41207] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:06,184 DEBUG [RS:2;jenkins-hbase4:42505] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:06,184 DEBUG [RS:2;jenkins-hbase4:42505] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42505,1676127604785' 2023-02-11 15:00:06,184 DEBUG [RS:2;jenkins-hbase4:42505] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 15:00:06,184 DEBUG [RS:0;jenkins-hbase4:41207] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41207,1676127604675' 2023-02-11 15:00:06,184 DEBUG [RS:0;jenkins-hbase4:41207] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 15:00:06,184 DEBUG [RS:1;jenkins-hbase4:33165] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 15:00:06,185 DEBUG [RS:2;jenkins-hbase4:42505] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 15:00:06,185 DEBUG [RS:0;jenkins-hbase4:41207] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 15:00:06,185 DEBUG [RS:1;jenkins-hbase4:33165] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 15:00:06,185 DEBUG [RS:2;jenkins-hbase4:42505] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 15:00:06,185 INFO [RS:1;jenkins-hbase4:33165] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 15:00:06,185 INFO [RS:2;jenkins-hbase4:42505] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 15:00:06,186 INFO [RS:1;jenkins-hbase4:33165] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 15:00:06,187 DEBUG [RS:0;jenkins-hbase4:41207] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 15:00:06,187 INFO [RS:2;jenkins-hbase4:42505] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 15:00:06,187 INFO [RS:0;jenkins-hbase4:41207] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 15:00:06,187 INFO [RS:0;jenkins-hbase4:41207] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 15:00:06,205 DEBUG [jenkins-hbase4:38517] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-11 15:00:06,209 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 15:00:06,214 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 15:00:06,214 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 15:00:06,214 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 15:00:06,214 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 15:00:06,217 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41207,1676127604675, state=OPENING 2023-02-11 15:00:06,225 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-11 15:00:06,226 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 15:00:06,227 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 15:00:06,230 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:06,297 INFO [RS:2;jenkins-hbase4:42505] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42505%2C1676127604785, suffix=, logDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,42505,1676127604785, archiveDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/oldWALs, maxLogs=32 2023-02-11 15:00:06,298 INFO [RS:0;jenkins-hbase4:41207] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41207%2C1676127604675, suffix=, logDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,41207,1676127604675, archiveDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/oldWALs, maxLogs=32 2023-02-11 15:00:06,297 INFO [RS:1;jenkins-hbase4:33165] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33165%2C1676127604743, suffix=, logDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,33165,1676127604743, archiveDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/oldWALs, maxLogs=32 2023-02-11 15:00:06,324 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45987,DS-a33290da-ad13-4ce6-ba63-edb151184900,DISK] 2023-02-11 15:00:06,324 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45987,DS-a33290da-ad13-4ce6-ba63-edb151184900,DISK] 2023-02-11 15:00:06,325 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38149,DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0,DISK] 2023-02-11 15:00:06,325 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34407,DS-27c45a68-a2cd-45c8-941a-921f2704e91e,DISK] 2023-02-11 15:00:06,340 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34407,DS-27c45a68-a2cd-45c8-941a-921f2704e91e,DISK] 2023-02-11 15:00:06,340 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38149,DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0,DISK] 2023-02-11 15:00:06,341 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38149,DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0,DISK] 2023-02-11 15:00:06,341 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34407,DS-27c45a68-a2cd-45c8-941a-921f2704e91e,DISK] 2023-02-11 15:00:06,342 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45987,DS-a33290da-ad13-4ce6-ba63-edb151184900,DISK] 2023-02-11 15:00:06,356 INFO [RS:1;jenkins-hbase4:33165] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,33165,1676127604743/jenkins-hbase4.apache.org%2C33165%2C1676127604743.1676127606305 2023-02-11 15:00:06,356 INFO [RS:2;jenkins-hbase4:42505] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,42505,1676127604785/jenkins-hbase4.apache.org%2C42505%2C1676127604785.1676127606304 2023-02-11 15:00:06,357 INFO [RS:0;jenkins-hbase4:41207] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,41207,1676127604675/jenkins-hbase4.apache.org%2C41207%2C1676127604675.1676127606305 2023-02-11 15:00:06,360 DEBUG [RS:1;jenkins-hbase4:33165] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45987,DS-a33290da-ad13-4ce6-ba63-edb151184900,DISK], DatanodeInfoWithStorage[127.0.0.1:34407,DS-27c45a68-a2cd-45c8-941a-921f2704e91e,DISK], DatanodeInfoWithStorage[127.0.0.1:38149,DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0,DISK]] 2023-02-11 15:00:06,360 DEBUG [RS:2;jenkins-hbase4:42505] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45987,DS-a33290da-ad13-4ce6-ba63-edb151184900,DISK], DatanodeInfoWithStorage[127.0.0.1:38149,DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0,DISK], DatanodeInfoWithStorage[127.0.0.1:34407,DS-27c45a68-a2cd-45c8-941a-921f2704e91e,DISK]] 2023-02-11 15:00:06,360 DEBUG [RS:0;jenkins-hbase4:41207] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34407,DS-27c45a68-a2cd-45c8-941a-921f2704e91e,DISK], DatanodeInfoWithStorage[127.0.0.1:38149,DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0,DISK], DatanodeInfoWithStorage[127.0.0.1:45987,DS-a33290da-ad13-4ce6-ba63-edb151184900,DISK]] 2023-02-11 15:00:06,414 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:06,416 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 15:00:06,420 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59200, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 15:00:06,433 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-11 15:00:06,433 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 15:00:06,437 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41207%2C1676127604675.meta, suffix=.meta, logDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,41207,1676127604675, archiveDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/oldWALs, maxLogs=32 2023-02-11 15:00:06,456 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45987,DS-a33290da-ad13-4ce6-ba63-edb151184900,DISK] 2023-02-11 15:00:06,458 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38149,DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0,DISK] 2023-02-11 15:00:06,459 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34407,DS-27c45a68-a2cd-45c8-941a-921f2704e91e,DISK] 2023-02-11 15:00:06,466 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/WALs/jenkins-hbase4.apache.org,41207,1676127604675/jenkins-hbase4.apache.org%2C41207%2C1676127604675.meta.1676127606438.meta 2023-02-11 15:00:06,467 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45987,DS-a33290da-ad13-4ce6-ba63-edb151184900,DISK], DatanodeInfoWithStorage[127.0.0.1:38149,DS-85c264b0-2b1c-42f5-b3b7-84196efbfad0,DISK], DatanodeInfoWithStorage[127.0.0.1:34407,DS-27c45a68-a2cd-45c8-941a-921f2704e91e,DISK]] 2023-02-11 15:00:06,467 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-11 15:00:06,469 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-11 15:00:06,491 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-11 15:00:06,497 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-11 15:00:06,502 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-11 15:00:06,502 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:06,502 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-11 15:00:06,502 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-11 15:00:06,505 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 15:00:06,507 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/info 2023-02-11 15:00:06,507 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/info 2023-02-11 15:00:06,508 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 15:00:06,509 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:06,509 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 15:00:06,510 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/rep_barrier 2023-02-11 15:00:06,510 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/rep_barrier 2023-02-11 15:00:06,511 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 15:00:06,512 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:06,512 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 15:00:06,514 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/table 2023-02-11 15:00:06,514 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/table 2023-02-11 15:00:06,514 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 15:00:06,515 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:06,517 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740 2023-02-11 15:00:06,519 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740 2023-02-11 15:00:06,523 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 15:00:06,525 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 15:00:06,527 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68068570, jitterRate=0.014300733804702759}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 15:00:06,527 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676127606503Writing region info on filesystem at 1676127606503Initializing all the Stores at 1676127606505 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676127606505Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676127606505Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676127606505Cleaning up temporary data from old regions at 1676127606521 (+16 ms)Cleaning up detritus from prior splits at 1676127606522 (+1 ms)Running coprocessor post-open hooks at 1676127606527 (+5 ms)Region opened successfully at 1676127606527 2023-02-11 15:00:06,540 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676127606406 2023-02-11 15:00:06,563 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-11 15:00:06,564 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-11 15:00:06,564 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41207,1676127604675, state=OPEN 2023-02-11 15:00:06,567 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-11 15:00:06,567 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 15:00:06,572 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-11 15:00:06,572 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41207,1676127604675 in 337 msec 2023-02-11 15:00:06,577 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-11 15:00:06,577 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 544 msec 2023-02-11 15:00:06,583 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 831 msec 2023-02-11 15:00:06,583 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676127604831, completionTime=-1 2023-02-11 15:00:06,583 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-11 15:00:06,583 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-11 15:00:06,651 DEBUG [hconnection-0x4553a206-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:06,654 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59210, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:06,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-11 15:00:06,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676127666672 2023-02-11 15:00:06,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676127726672 2023-02-11 15:00:06,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 89 msec 2023-02-11 15:00:06,700 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38517,1676127603486-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,701 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38517,1676127603486-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,701 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38517,1676127603486-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,702 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:38517, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,703 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-11 15:00:06,709 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-11 15:00:06,723 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-11 15:00:06,724 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 15:00:06,734 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-11 15:00:06,737 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 15:00:06,739 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 15:00:06,761 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:06,763 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5 empty. 2023-02-11 15:00:06,764 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:06,764 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-11 15:00:06,807 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-11 15:00:06,810 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => a3f59341dd39a469cff752a69644f1c5, NAME => 'hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:06,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:06,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing a3f59341dd39a469cff752a69644f1c5, disabling compactions & flushes 2023-02-11 15:00:06,833 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:06,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:06,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. after waiting 0 ms 2023-02-11 15:00:06,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:06,833 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:06,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for a3f59341dd39a469cff752a69644f1c5: Waiting for close lock at 1676127606833Disabling compacts and flushes for region at 1676127606833Disabling writes for close at 1676127606833Writing region close event to WAL at 1676127606833Closed at 1676127606833 2023-02-11 15:00:06,838 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 15:00:06,854 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676127606841"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127606841"}]},"ts":"1676127606841"} 2023-02-11 15:00:06,881 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-11 15:00:06,883 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 15:00:06,888 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127606883"}]},"ts":"1676127606883"} 2023-02-11 15:00:06,894 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-11 15:00:06,898 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 15:00:06,899 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 15:00:06,899 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 15:00:06,899 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 15:00:06,899 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 15:00:06,903 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a3f59341dd39a469cff752a69644f1c5, ASSIGN}] 2023-02-11 15:00:06,908 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a3f59341dd39a469cff752a69644f1c5, ASSIGN 2023-02-11 15:00:06,910 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=a3f59341dd39a469cff752a69644f1c5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=false 2023-02-11 15:00:07,062 INFO [jenkins-hbase4:38517] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-11 15:00:07,063 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a3f59341dd39a469cff752a69644f1c5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:07,063 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676127607063"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127607063"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127607063"}]},"ts":"1676127607063"} 2023-02-11 15:00:07,068 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure a3f59341dd39a469cff752a69644f1c5, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:07,228 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:07,228 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a3f59341dd39a469cff752a69644f1c5, NAME => 'hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5.', STARTKEY => '', ENDKEY => ''} 2023-02-11 15:00:07,230 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:07,230 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:07,230 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:07,230 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:07,232 INFO [StoreOpener-a3f59341dd39a469cff752a69644f1c5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:07,234 DEBUG [StoreOpener-a3f59341dd39a469cff752a69644f1c5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5/info 2023-02-11 15:00:07,234 DEBUG [StoreOpener-a3f59341dd39a469cff752a69644f1c5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5/info 2023-02-11 15:00:07,235 INFO [StoreOpener-a3f59341dd39a469cff752a69644f1c5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a3f59341dd39a469cff752a69644f1c5 columnFamilyName info 2023-02-11 15:00:07,235 INFO [StoreOpener-a3f59341dd39a469cff752a69644f1c5-1] regionserver.HStore(310): Store=a3f59341dd39a469cff752a69644f1c5/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:07,237 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:07,238 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:07,248 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:07,252 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:07,253 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a3f59341dd39a469cff752a69644f1c5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62948012, jitterRate=-0.06200152635574341}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:07,253 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a3f59341dd39a469cff752a69644f1c5: Running coprocessor pre-open hook at 1676127607230Writing region info on filesystem at 1676127607230Initializing all the Stores at 1676127607231 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676127607232 (+1 ms)Cleaning up temporary data from old regions at 1676127607241 (+9 ms)Cleaning up detritus from prior splits at 1676127607241Running coprocessor post-open hooks at 1676127607253 (+12 ms)Region opened successfully at 1676127607253 2023-02-11 15:00:07,255 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5., pid=6, masterSystemTime=1676127607221 2023-02-11 15:00:07,258 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:07,258 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:07,260 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a3f59341dd39a469cff752a69644f1c5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:07,261 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676127607259"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127607259"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127607259"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127607259"}]},"ts":"1676127607259"} 2023-02-11 15:00:07,268 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-11 15:00:07,268 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure a3f59341dd39a469cff752a69644f1c5, server=jenkins-hbase4.apache.org,41207,1676127604675 in 196 msec 2023-02-11 15:00:07,272 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-11 15:00:07,272 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=a3f59341dd39a469cff752a69644f1c5, ASSIGN in 365 msec 2023-02-11 15:00:07,274 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 15:00:07,274 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127607274"}]},"ts":"1676127607274"} 2023-02-11 15:00:07,276 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-11 15:00:07,280 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 15:00:07,283 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 555 msec 2023-02-11 15:00:07,337 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-11 15:00:07,339 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-11 15:00:07,339 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 15:00:07,374 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-11 15:00:07,393 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 15:00:07,400 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 33 msec 2023-02-11 15:00:07,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-11 15:00:07,420 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 15:00:07,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 20 msec 2023-02-11 15:00:07,443 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-11 15:00:07,447 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-11 15:00:07,447 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.602sec 2023-02-11 15:00:07,450 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-11 15:00:07,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-11 15:00:07,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-11 15:00:07,453 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38517,1676127603486-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-11 15:00:07,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38517,1676127603486-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-11 15:00:07,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-11 15:00:07,516 DEBUG [Listener at localhost/39605] zookeeper.ReadOnlyZKClient(139): Connect 0x1c633018 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:07,521 DEBUG [Listener at localhost/39605] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b210801, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:07,538 DEBUG [hconnection-0x7b9b1635-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:07,550 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59222, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:07,563 INFO [Listener at localhost/39605] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,38517,1676127603486 2023-02-11 15:00:07,595 INFO [Listener at localhost/39605] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=359, OpenFileDescriptor=691, MaxFileDescriptor=60000, SystemLoadAverage=259, ProcessCount=171, AvailableMemoryMB=3688 2023-02-11 15:00:07,609 DEBUG [Listener at localhost/39605] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-11 15:00:07,646 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41390, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-11 15:00:07,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-11 15:00:07,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:07,665 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676127607597 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 15:00:07,667 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676127607597 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 15:00:07,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676127607597" procId is: 9 2023-02-11 15:00:07,676 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:07,676 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:07,677 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:07,676 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:07,677 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:07,677 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 empty. 2023-02-11 15:00:07,676 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:07,678 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b empty. 2023-02-11 15:00:07,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 15:00:07,680 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 empty. 2023-02-11 15:00:07,680 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f empty. 2023-02-11 15:00:07,680 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:07,681 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 empty. 2023-02-11 15:00:07,681 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e empty. 2023-02-11 15:00:07,681 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:07,682 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:07,682 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:07,682 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:07,681 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:07,686 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676127607597 regions 2023-02-11 15:00:07,708 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000001 2023-02-11 15:00:07,710 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(7671): creating {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:07,712 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(7671): creating {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:07,713 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(7671): creating {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:07,762 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:07,764 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:07,766 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:07,766 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1603): Closing 704f07b3665f9baa8c5a5484e3bdbcd6, disabling compactions & flushes 2023-02-11 15:00:07,766 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1603): Closing 7677bc32f6d3f7225f4cfdeb2068fc26, disabling compactions & flushes 2023-02-11 15:00:07,767 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:07,766 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1603): Closing cd2163f692eb299f43cb55be4d4dc49f, disabling compactions & flushes 2023-02-11 15:00:07,767 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:07,767 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:07,767 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. after waiting 0 ms 2023-02-11 15:00:07,767 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:07,767 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:07,767 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:07,767 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:07,767 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. after waiting 0 ms 2023-02-11 15:00:07,767 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1557): Region close journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Waiting for close lock at 1676127607766Disabling compacts and flushes for region at 1676127607766Disabling writes for close at 1676127607767 (+1 ms)Writing region close event to WAL at 1676127607767Closed at 1676127607767 2023-02-11 15:00:07,767 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:07,767 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:07,767 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:07,767 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1557): Region close journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Waiting for close lock at 1676127607766Disabling compacts and flushes for region at 1676127607766Disabling writes for close at 1676127607767 (+1 ms)Writing region close event to WAL at 1676127607767Closed at 1676127607767 2023-02-11 15:00:07,768 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. after waiting 0 ms 2023-02-11 15:00:07,768 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:07,768 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(7671): creating {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:07,768 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(7671): creating {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:07,768 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:07,769 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1557): Region close journal for cd2163f692eb299f43cb55be4d4dc49f: Waiting for close lock at 1676127607766Disabling compacts and flushes for region at 1676127607766Disabling writes for close at 1676127607768 (+2 ms)Writing region close event to WAL at 1676127607768Closed at 1676127607768 2023-02-11 15:00:07,769 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(7671): creating {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:07,799 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:07,799 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:07,801 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1603): Closing 78f87f7851c87dde38a4d94bc805db9b, disabling compactions & flushes 2023-02-11 15:00:07,801 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1603): Closing 94a0d6907c18dd6eadf539b9aa0231e2, disabling compactions & flushes 2023-02-11 15:00:07,801 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:07,801 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:07,801 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. after waiting 0 ms 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. after waiting 0 ms 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1603): Closing 2c301b6be53a2e3d2c7174a30417ca8e, disabling compactions & flushes 2023-02-11 15:00:07,802 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-1] regionserver.HRegion(1557): Region close journal for 78f87f7851c87dde38a4d94bc805db9b: Waiting for close lock at 1676127607801Disabling compacts and flushes for region at 1676127607801Disabling writes for close at 1676127607802 (+1 ms)Writing region close event to WAL at 1676127607802Closed at 1676127607802 2023-02-11 15:00:07,802 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:07,802 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-0] regionserver.HRegion(1557): Region close journal for 94a0d6907c18dd6eadf539b9aa0231e2: Waiting for close lock at 1676127607801Disabling compacts and flushes for region at 1676127607801Disabling writes for close at 1676127607802 (+1 ms)Writing region close event to WAL at 1676127607802Closed at 1676127607802 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. after waiting 0 ms 2023-02-11 15:00:07,802 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:07,802 INFO [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:07,803 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676127607597-pool-2] regionserver.HRegion(1557): Region close journal for 2c301b6be53a2e3d2c7174a30417ca8e: Waiting for close lock at 1676127607802Disabling compacts and flushes for region at 1676127607802Disabling writes for close at 1676127607802Writing region close event to WAL at 1676127607802Closed at 1676127607802 2023-02-11 15:00:07,807 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676127607597 execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 15:00:07,809 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127607808"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127607808"}]},"ts":"1676127607808"} 2023-02-11 15:00:07,809 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127607808"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127607808"}]},"ts":"1676127607808"} 2023-02-11 15:00:07,809 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127607808"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127607808"}]},"ts":"1676127607808"} 2023-02-11 15:00:07,809 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127607808"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127607808"}]},"ts":"1676127607808"} 2023-02-11 15:00:07,809 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127607808"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127607808"}]},"ts":"1676127607808"} 2023-02-11 15:00:07,809 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127607808"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127607808"}]},"ts":"1676127607808"} 2023-02-11 15:00:07,855 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 15:00:07,857 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676127607597 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 15:00:07,857 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127607857"}]},"ts":"1676127607857"} 2023-02-11 15:00:07,859 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLING in hbase:meta 2023-02-11 15:00:07,864 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 15:00:07,868 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 15:00:07,868 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 15:00:07,868 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 15:00:07,868 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 15:00:07,869 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN}] 2023-02-11 15:00:07,872 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN 2023-02-11 15:00:07,872 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN 2023-02-11 15:00:07,873 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN 2023-02-11 15:00:07,873 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN 2023-02-11 15:00:07,875 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=false 2023-02-11 15:00:07,875 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=false 2023-02-11 15:00:07,875 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN 2023-02-11 15:00:07,875 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=false 2023-02-11 15:00:07,875 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=false 2023-02-11 15:00:07,876 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN 2023-02-11 15:00:07,876 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=false 2023-02-11 15:00:07,878 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=false 2023-02-11 15:00:07,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 15:00:08,025 INFO [jenkins-hbase4:38517] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 15:00:08,028 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:08,028 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:08,028 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:08,028 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127608028"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608028"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608028"}]},"ts":"1676127608028"} 2023-02-11 15:00:08,028 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:08,029 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127608028"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608028"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608028"}]},"ts":"1676127608028"} 2023-02-11 15:00:08,028 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:08,029 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608028"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608028"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608028"}]},"ts":"1676127608028"} 2023-02-11 15:00:08,029 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608028"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608028"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608028"}]},"ts":"1676127608028"} 2023-02-11 15:00:08,029 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608028"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608028"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608028"}]},"ts":"1676127608028"} 2023-02-11 15:00:08,032 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=10, state=RUNNABLE; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:08,034 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=15, state=RUNNABLE; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:08,036 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=11, state=RUNNABLE; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:08,038 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=14, state=RUNNABLE; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:08,040 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=12, state=RUNNABLE; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:08,043 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:08,043 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608043"}]},"ts":"1676127608043"} 2023-02-11 15:00:08,059 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=13, state=RUNNABLE; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:08,186 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:08,186 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 15:00:08,190 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48484, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 15:00:08,193 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:08,194 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 15:00:08,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:08,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 15:00:08,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 15:00:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:08,199 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42474, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 15:00:08,202 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:08,206 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:08,208 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:08,208 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:08,208 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:08,208 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:08,212 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7677bc32f6d3f7225f4cfdeb2068fc26 columnFamilyName cf 2023-02-11 15:00:08,212 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 704f07b3665f9baa8c5a5484e3bdbcd6 columnFamilyName cf 2023-02-11 15:00:08,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:08,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 15:00:08,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:08,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:08,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:08,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:08,217 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(310): Store=704f07b3665f9baa8c5a5484e3bdbcd6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:08,217 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(310): Store=7677bc32f6d3f7225f4cfdeb2068fc26/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:08,223 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:08,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:08,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:08,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:08,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:08,226 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:08,227 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:08,227 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cd2163f692eb299f43cb55be4d4dc49f columnFamilyName cf 2023-02-11 15:00:08,229 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(310): Store=cd2163f692eb299f43cb55be4d4dc49f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:08,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:08,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:08,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:08,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:08,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:08,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:08,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:08,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:08,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened cd2163f692eb299f43cb55be4d4dc49f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64786102, jitterRate=-0.03461185097694397}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:08,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7677bc32f6d3f7225f4cfdeb2068fc26; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69491847, jitterRate=0.03550921380519867}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:08,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for cd2163f692eb299f43cb55be4d4dc49f: Running coprocessor pre-open hook at 1676127608216Writing region info on filesystem at 1676127608216Initializing all the Stores at 1676127608218 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127608218Cleaning up temporary data from old regions at 1676127608234 (+16 ms)Cleaning up detritus from prior splits at 1676127608235 (+1 ms)Running coprocessor post-open hooks at 1676127608249 (+14 ms)Region opened successfully at 1676127608249 2023-02-11 15:00:08,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Running coprocessor pre-open hook at 1676127608198Writing region info on filesystem at 1676127608198Initializing all the Stores at 1676127608200 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127608200Cleaning up temporary data from old regions at 1676127608226 (+26 ms)Cleaning up detritus from prior splits at 1676127608227 (+1 ms)Running coprocessor post-open hooks at 1676127608249 (+22 ms)Region opened successfully at 1676127608249 2023-02-11 15:00:08,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 704f07b3665f9baa8c5a5484e3bdbcd6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75248045, jitterRate=0.12128324806690216}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:08,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Running coprocessor pre-open hook at 1676127608199Writing region info on filesystem at 1676127608199Initializing all the Stores at 1676127608201 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127608201Cleaning up temporary data from old regions at 1676127608226 (+25 ms)Cleaning up detritus from prior splits at 1676127608227 (+1 ms)Running coprocessor post-open hooks at 1676127608250 (+23 ms)Region opened successfully at 1676127608250 2023-02-11 15:00:08,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26., pid=16, masterSystemTime=1676127608186 2023-02-11 15:00:08,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6., pid=20, masterSystemTime=1676127608189 2023-02-11 15:00:08,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f., pid=18, masterSystemTime=1676127608193 2023-02-11 15:00:08,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:08,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:08,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:08,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 15:00:08,258 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:08,259 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608258"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127608258"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127608258"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127608258"}]},"ts":"1676127608258"} 2023-02-11 15:00:08,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:08,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:08,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:08,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:08,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:08,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:08,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:08,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 15:00:08,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:08,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:08,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:08,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:08,269 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:08,269 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:08,271 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:08,271 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127608270"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127608270"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127608270"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127608270"}]},"ts":"1676127608270"} 2023-02-11 15:00:08,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:08,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:08,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:08,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 15:00:08,275 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:08,277 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:08,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:08,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:08,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:08,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:08,276 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:08,278 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:08,277 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94a0d6907c18dd6eadf539b9aa0231e2 columnFamilyName cf 2023-02-11 15:00:08,277 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:08,279 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78f87f7851c87dde38a4d94bc805db9b columnFamilyName cf 2023-02-11 15:00:08,280 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608276"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127608276"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127608276"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127608276"}]},"ts":"1676127608276"} 2023-02-11 15:00:08,281 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=12 2023-02-11 15:00:08,286 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=12, state=SUCCESS; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675 in 232 msec 2023-02-11 15:00:08,283 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(310): Store=94a0d6907c18dd6eadf539b9aa0231e2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:08,282 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(310): Store=78f87f7851c87dde38a4d94bc805db9b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:08,287 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN in 413 msec 2023-02-11 15:00:08,286 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:08,286 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=10 2023-02-11 15:00:08,289 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=10, state=SUCCESS; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785 in 245 msec 2023-02-11 15:00:08,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:08,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:08,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:08,292 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:08,292 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:08,292 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=11 2023-02-11 15:00:08,292 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=11, state=SUCCESS; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743 in 249 msec 2023-02-11 15:00:08,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:08,293 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c301b6be53a2e3d2c7174a30417ca8e columnFamilyName cf 2023-02-11 15:00:08,293 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN in 417 msec 2023-02-11 15:00:08,295 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(310): Store=2c301b6be53a2e3d2c7174a30417ca8e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:08,296 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN in 423 msec 2023-02-11 15:00:08,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:08,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:08,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:08,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:08,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:08,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 94a0d6907c18dd6eadf539b9aa0231e2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74499427, jitterRate=0.11012797057628632}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:08,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 94a0d6907c18dd6eadf539b9aa0231e2: Running coprocessor pre-open hook at 1676127608264Writing region info on filesystem at 1676127608264Initializing all the Stores at 1676127608265 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127608266 (+1 ms)Cleaning up temporary data from old regions at 1676127608292 (+26 ms)Cleaning up detritus from prior splits at 1676127608293 (+1 ms)Running coprocessor post-open hooks at 1676127608304 (+11 ms)Region opened successfully at 1676127608304 2023-02-11 15:00:08,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:08,306 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2., pid=19, masterSystemTime=1676127608186 2023-02-11 15:00:08,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 78f87f7851c87dde38a4d94bc805db9b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60177000, jitterRate=-0.10329282283782959}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:08,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 78f87f7851c87dde38a4d94bc805db9b: Running coprocessor pre-open hook at 1676127608259Writing region info on filesystem at 1676127608259Initializing all the Stores at 1676127608264 (+5 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127608264Cleaning up temporary data from old regions at 1676127608293 (+29 ms)Cleaning up detritus from prior splits at 1676127608295 (+2 ms)Running coprocessor post-open hooks at 1676127608307 (+12 ms)Region opened successfully at 1676127608307 2023-02-11 15:00:08,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:08,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b., pid=17, masterSystemTime=1676127608189 2023-02-11 15:00:08,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:08,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:08,313 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:08,313 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608312"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127608312"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127608312"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127608312"}]},"ts":"1676127608312"} 2023-02-11 15:00:08,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:08,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:08,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:08,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2c301b6be53a2e3d2c7174a30417ca8e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73781960, jitterRate=0.09943687915802002}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:08,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2c301b6be53a2e3d2c7174a30417ca8e: Running coprocessor pre-open hook at 1676127608278Writing region info on filesystem at 1676127608278Initializing all the Stores at 1676127608281 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127608281Cleaning up temporary data from old regions at 1676127608300 (+19 ms)Cleaning up detritus from prior splits at 1676127608303 (+3 ms)Running coprocessor post-open hooks at 1676127608315 (+12 ms)Region opened successfully at 1676127608315 2023-02-11 15:00:08,316 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:08,317 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127608316"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127608316"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127608316"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127608316"}]},"ts":"1676127608316"} 2023-02-11 15:00:08,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e., pid=21, masterSystemTime=1676127608193 2023-02-11 15:00:08,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:08,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:08,321 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:08,322 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=14 2023-02-11 15:00:08,322 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=14, state=SUCCESS; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785 in 279 msec 2023-02-11 15:00:08,322 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608321"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127608321"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127608321"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127608321"}]},"ts":"1676127608321"} 2023-02-11 15:00:08,326 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN in 453 msec 2023-02-11 15:00:08,327 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=15 2023-02-11 15:00:08,328 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=15, state=SUCCESS; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675 in 287 msec 2023-02-11 15:00:08,330 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=13 2023-02-11 15:00:08,331 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=13, state=SUCCESS; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743 in 266 msec 2023-02-11 15:00:08,331 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN in 458 msec 2023-02-11 15:00:08,334 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=9 2023-02-11 15:00:08,334 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN in 461 msec 2023-02-11 15:00:08,335 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676127607597 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 15:00:08,335 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127608335"}]},"ts":"1676127608335"} 2023-02-11 15:00:08,338 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLED in hbase:meta 2023-02-11 15:00:08,342 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676127607597 execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 15:00:08,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676127607597 in 682 msec 2023-02-11 15:00:08,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 15:00:08,438 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 9 completed 2023-02-11 15:00:08,446 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676127607597 2023-02-11 15:00:08,447 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:08,451 INFO [Listener at localhost/39605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 15:00:08,473 DEBUG [hconnection-0x7b9b1635-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:08,476 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48492, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:08,479 DEBUG [hconnection-0x7b9b1635-shared-pool-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:08,484 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42476, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:08,497 INFO [Listener at localhost/39605] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:08,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676127607597 2023-02-11 15:00:08,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:08,512 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127608512"}]},"ts":"1676127608512"} 2023-02-11 15:00:08,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-11 15:00:08,514 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLING in hbase:meta 2023-02-11 15:00:08,516 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676127607597 to state=DISABLING 2023-02-11 15:00:08,518 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN}] 2023-02-11 15:00:08,521 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN 2023-02-11 15:00:08,521 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN 2023-02-11 15:00:08,521 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN 2023-02-11 15:00:08,521 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN 2023-02-11 15:00:08,521 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN 2023-02-11 15:00:08,522 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:08,522 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127608522"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608522"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608522"}]},"ts":"1676127608522"} 2023-02-11 15:00:08,523 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:08,523 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:08,523 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:08,523 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:08,523 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608523"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608523"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608523"}]},"ts":"1676127608523"} 2023-02-11 15:00:08,523 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127608523"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608523"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608523"}]},"ts":"1676127608523"} 2023-02-11 15:00:08,523 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608523"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608523"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608523"}]},"ts":"1676127608523"} 2023-02-11 15:00:08,523 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608523"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608523"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608523"}]},"ts":"1676127608523"} 2023-02-11 15:00:08,525 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=23, state=RUNNABLE; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:08,527 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=25, state=RUNNABLE; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:08,529 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:08,531 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=24, state=RUNNABLE; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:08,533 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=27, state=RUNNABLE; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:08,539 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN 2023-02-11 15:00:08,540 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:08,541 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608540"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127608540"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127608540"}]},"ts":"1676127608540"} 2023-02-11 15:00:08,543 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=26, state=RUNNABLE; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:08,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:08,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:08,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 94a0d6907c18dd6eadf539b9aa0231e2, disabling compactions & flushes 2023-02-11 15:00:08,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:08,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing cd2163f692eb299f43cb55be4d4dc49f, disabling compactions & flushes 2023-02-11 15:00:08,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:08,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 78f87f7851c87dde38a4d94bc805db9b, disabling compactions & flushes 2023-02-11 15:00:08,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:08,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:08,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:08,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:08,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. after waiting 0 ms 2023-02-11 15:00:08,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. after waiting 0 ms 2023-02-11 15:00:08,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:08,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:08,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. after waiting 0 ms 2023-02-11 15:00:08,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:08,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:08,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:08,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:08,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:08,704 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:08,704 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:08,704 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 78f87f7851c87dde38a4d94bc805db9b: Waiting for close lock at 1676127608691Running coprocessor pre-close hooks at 1676127608691Disabling compacts and flushes for region at 1676127608691Disabling writes for close at 1676127608692 (+1 ms)Writing region close event to WAL at 1676127608694 (+2 ms)Running coprocessor post-close hooks at 1676127608704 (+10 ms)Closed at 1676127608704 2023-02-11 15:00:08,704 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for cd2163f692eb299f43cb55be4d4dc49f: Waiting for close lock at 1676127608690Running coprocessor pre-close hooks at 1676127608690Disabling compacts and flushes for region at 1676127608690Disabling writes for close at 1676127608692 (+2 ms)Writing region close event to WAL at 1676127608693 (+1 ms)Running coprocessor post-close hooks at 1676127608704 (+11 ms)Closed at 1676127608704 2023-02-11 15:00:08,704 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:08,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 94a0d6907c18dd6eadf539b9aa0231e2: Waiting for close lock at 1676127608690Running coprocessor pre-close hooks at 1676127608690Disabling compacts and flushes for region at 1676127608690Disabling writes for close at 1676127608692 (+2 ms)Writing region close event to WAL at 1676127608693 (+1 ms)Running coprocessor post-close hooks at 1676127608704 (+11 ms)Closed at 1676127608704 2023-02-11 15:00:08,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:08,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:08,707 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 704f07b3665f9baa8c5a5484e3bdbcd6, disabling compactions & flushes 2023-02-11 15:00:08,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:08,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:08,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. after waiting 0 ms 2023-02-11 15:00:08,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:08,709 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSED 2023-02-11 15:00:08,709 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127608709"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127608709"}]},"ts":"1676127608709"} 2023-02-11 15:00:08,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:08,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:08,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2c301b6be53a2e3d2c7174a30417ca8e, disabling compactions & flushes 2023-02-11 15:00:08,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:08,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:08,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. after waiting 0 ms 2023-02-11 15:00:08,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:08,711 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSED 2023-02-11 15:00:08,714 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608711"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127608711"}]},"ts":"1676127608711"} 2023-02-11 15:00:08,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:08,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:08,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7677bc32f6d3f7225f4cfdeb2068fc26, disabling compactions & flushes 2023-02-11 15:00:08,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:08,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:08,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. after waiting 0 ms 2023-02-11 15:00:08,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:08,717 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSED 2023-02-11 15:00:08,717 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608717"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127608717"}]},"ts":"1676127608717"} 2023-02-11 15:00:08,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:08,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:08,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Waiting for close lock at 1676127608707Running coprocessor pre-close hooks at 1676127608707Disabling compacts and flushes for region at 1676127608707Disabling writes for close at 1676127608708 (+1 ms)Writing region close event to WAL at 1676127608708Running coprocessor post-close hooks at 1676127608721 (+13 ms)Closed at 1676127608721 2023-02-11 15:00:08,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=28 2023-02-11 15:00:08,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:08,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675 in 187 msec 2023-02-11 15:00:08,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:08,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:08,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2c301b6be53a2e3d2c7174a30417ca8e: Waiting for close lock at 1676127608709Running coprocessor pre-close hooks at 1676127608709Disabling compacts and flushes for region at 1676127608709Disabling writes for close at 1676127608711 (+2 ms)Writing region close event to WAL at 1676127608711Running coprocessor post-close hooks at 1676127608724 (+13 ms)Closed at 1676127608724 2023-02-11 15:00:08,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=24 2023-02-11 15:00:08,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=24, state=SUCCESS; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743 in 189 msec 2023-02-11 15:00:08,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:08,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Waiting for close lock at 1676127608715Running coprocessor pre-close hooks at 1676127608715Disabling compacts and flushes for region at 1676127608715Disabling writes for close at 1676127608715Writing region close event to WAL at 1676127608715Running coprocessor post-close hooks at 1676127608724 (+9 ms)Closed at 1676127608724 2023-02-11 15:00:08,729 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN in 204 msec 2023-02-11 15:00:08,729 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSED 2023-02-11 15:00:08,729 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608729"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127608729"}]},"ts":"1676127608729"} 2023-02-11 15:00:08,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:08,730 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=27 2023-02-11 15:00:08,731 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=27, state=SUCCESS; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785 in 188 msec 2023-02-11 15:00:08,731 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSED 2023-02-11 15:00:08,731 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN in 206 msec 2023-02-11 15:00:08,731 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127608731"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127608731"}]},"ts":"1676127608731"} 2023-02-11 15:00:08,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:08,733 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSED 2023-02-11 15:00:08,733 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127608733"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127608733"}]},"ts":"1676127608733"} 2023-02-11 15:00:08,733 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:08,734 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN in 213 msec 2023-02-11 15:00:08,737 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=25 2023-02-11 15:00:08,737 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=25, state=SUCCESS; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675 in 206 msec 2023-02-11 15:00:08,739 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=26 2023-02-11 15:00:08,739 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=26, state=SUCCESS; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743 in 192 msec 2023-02-11 15:00:08,739 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN in 219 msec 2023-02-11 15:00:08,740 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=23 2023-02-11 15:00:08,740 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=23, state=SUCCESS; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785 in 211 msec 2023-02-11 15:00:08,741 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN in 221 msec 2023-02-11 15:00:08,742 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-02-11 15:00:08,742 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN in 222 msec 2023-02-11 15:00:08,743 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127608743"}]},"ts":"1676127608743"} 2023-02-11 15:00:08,745 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLED in hbase:meta 2023-02-11 15:00:08,747 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676127607597 to state=DISABLED 2023-02-11 15:00:08,750 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676127607597 in 246 msec 2023-02-11 15:00:08,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-11 15:00:08,765 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 22 completed 2023-02-11 15:00:08,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:08,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676127608775 (current time:1676127608775). 2023-02-11 15:00:08,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 15:00:08,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(139): Connect 0x7b9c262e to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:08,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d6ae7d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:08,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:08,789 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59228, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:08,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7b9c262e to 127.0.0.1:58940 2023-02-11 15:00:08,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:08,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 15:00:08,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:08,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:08,818 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:08,819 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE; timeout=600000, timestamp=1676128208819 2023-02-11 15:00:08,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:08,819 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676127607597 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 2023-02-11 15:00:08,824 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 15:00:08,825 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 15:00:08,839 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } 2023-02-11 15:00:08,843 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:08,843 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:08,843 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:08,843 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:08,843 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:08,843 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:08,843 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:08,848 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:08,848 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:08,848 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:08,848 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:08,848 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:08,848 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:08,848 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:08,855 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:08,855 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:08,855 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:08,855 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:08,855 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:08,855 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:08,856 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:08,923 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } as finished. 2023-02-11 15:00:08,924 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676127607597 2023-02-11 15:00:08,926 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/emptySnaptb-1676127607597 2023-02-11 15:00:08,967 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/emptySnaptb-1676127607597 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/emptySnaptb-1676127607597 2023-02-11 15:00:09,012 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676127607597 of table testRestoreSnapshot-1676127607597 completed 2023-02-11 15:00:09,012 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/emptySnaptb-1676127607597 2023-02-11 15:00:09,012 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/emptySnaptb-1676127607597 2023-02-11 15:00:09,012 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676127607597 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 at 1676127608820Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } at 1676127608839 (+19 ms)Consolidate snapshot: emptySnaptb-1676127607597 at 1676127608923 (+84 ms)Loading Region manifests for emptySnaptb-1676127607597 at 1676127608924 (+1 ms)Writing data manifest for emptySnaptb-1676127607597 at 1676127608939 (+15 ms)Verifying snapshot: emptySnaptb-1676127607597 at 1676127608954 (+15 ms)Snapshot emptySnaptb-1676127607597 of table testRestoreSnapshot-1676127607597 completed at 1676127609012 (+58 ms) 2023-02-11 15:00:09,014 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:09,016 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE in 207 msec 2023-02-11 15:00:09,075 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 15:00:09,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } is done 2023-02-11 15:00:09,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 15:00:09,081 INFO [Listener at localhost/39605] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:09,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676127607597 2023-02-11 15:00:09,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:09,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 15:00:09,097 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676127607597 2023-02-11 15:00:09,097 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127609097"}]},"ts":"1676127609097"} 2023-02-11 15:00:09,100 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLING in hbase:meta 2023-02-11 15:00:09,103 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 15:00:09,108 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN}] 2023-02-11 15:00:09,111 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN 2023-02-11 15:00:09,111 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN 2023-02-11 15:00:09,111 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN 2023-02-11 15:00:09,111 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN 2023-02-11 15:00:09,113 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN 2023-02-11 15:00:09,113 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:09,113 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:09,113 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=true 2023-02-11 15:00:09,113 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=true 2023-02-11 15:00:09,113 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:09,113 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=true 2023-02-11 15:00:09,113 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:09,114 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=true 2023-02-11 15:00:09,115 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:09,115 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=true 2023-02-11 15:00:09,116 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN 2023-02-11 15:00:09,117 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:09,117 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=true 2023-02-11 15:00:09,263 INFO [jenkins-hbase4:38517] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 15:00:09,264 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:09,264 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:09,264 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:09,264 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127609264"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609264"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609264"}]},"ts":"1676127609264"} 2023-02-11 15:00:09,264 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:09,264 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:09,264 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609264"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609264"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609264"}]},"ts":"1676127609264"} 2023-02-11 15:00:09,264 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609264"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609264"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609264"}]},"ts":"1676127609264"} 2023-02-11 15:00:09,264 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127609264"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609264"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609264"}]},"ts":"1676127609264"} 2023-02-11 15:00:09,264 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609264"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609264"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609264"}]},"ts":"1676127609264"} 2023-02-11 15:00:09,267 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:09,268 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=39, state=RUNNABLE; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:09,269 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=38, state=RUNNABLE; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:09,270 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=37, state=RUNNABLE; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:09,271 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=41, state=RUNNABLE; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:09,273 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:09,273 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609273"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609273"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609273"}]},"ts":"1676127609273"} 2023-02-11 15:00:09,275 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=40, state=RUNNABLE; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:09,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 15:00:09,425 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:09,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 15:00:09,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:09,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:09,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:09,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:09,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:09,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 15:00:09,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:09,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:09,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:09,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:09,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:09,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 15:00:09,428 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:09,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:09,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:09,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:09,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:09,429 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:09,430 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:09,430 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:09,430 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:09,430 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78f87f7851c87dde38a4d94bc805db9b columnFamilyName cf 2023-02-11 15:00:09,431 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:09,431 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:09,432 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:09,432 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:09,432 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(310): Store=78f87f7851c87dde38a4d94bc805db9b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:09,432 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cd2163f692eb299f43cb55be4d4dc49f columnFamilyName cf 2023-02-11 15:00:09,433 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7677bc32f6d3f7225f4cfdeb2068fc26 columnFamilyName cf 2023-02-11 15:00:09,435 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(310): Store=cd2163f692eb299f43cb55be4d4dc49f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:09,435 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(310): Store=7677bc32f6d3f7225f4cfdeb2068fc26/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:09,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:09,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:09,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:09,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:09,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:09,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:09,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:09,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:09,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:09,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 78f87f7851c87dde38a4d94bc805db9b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67414937, jitterRate=0.004560843110084534}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:09,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 78f87f7851c87dde38a4d94bc805db9b: Running coprocessor pre-open hook at 1676127609426Writing region info on filesystem at 1676127609426Initializing all the Stores at 1676127609427 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127609428 (+1 ms)Cleaning up temporary data from old regions at 1676127609438 (+10 ms)Cleaning up detritus from prior splits at 1676127609438Running coprocessor post-open hooks at 1676127609444 (+6 ms)Region opened successfully at 1676127609444 2023-02-11 15:00:09,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7677bc32f6d3f7225f4cfdeb2068fc26; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73245073, jitterRate=0.09143663942813873}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:09,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Running coprocessor pre-open hook at 1676127609428Writing region info on filesystem at 1676127609428Initializing all the Stores at 1676127609430 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127609430Cleaning up temporary data from old regions at 1676127609439 (+9 ms)Cleaning up detritus from prior splits at 1676127609440 (+1 ms)Running coprocessor post-open hooks at 1676127609445 (+5 ms)Region opened successfully at 1676127609445 2023-02-11 15:00:09,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened cd2163f692eb299f43cb55be4d4dc49f; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63121121, jitterRate=-0.059422001242637634}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:09,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for cd2163f692eb299f43cb55be4d4dc49f: Running coprocessor pre-open hook at 1676127609428Writing region info on filesystem at 1676127609428Initializing all the Stores at 1676127609429 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127609429Cleaning up temporary data from old regions at 1676127609439 (+10 ms)Cleaning up detritus from prior splits at 1676127609440 (+1 ms)Running coprocessor post-open hooks at 1676127609445 (+5 ms)Region opened successfully at 1676127609445 2023-02-11 15:00:09,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b., pid=43, masterSystemTime=1676127609420 2023-02-11 15:00:09,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26., pid=46, masterSystemTime=1676127609424 2023-02-11 15:00:09,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f., pid=45, masterSystemTime=1676127609422 2023-02-11 15:00:09,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:09,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:09,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:09,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 15:00:09,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:09,449 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:09,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:09,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:09,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:09,449 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127609449"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127609449"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127609449"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127609449"}]},"ts":"1676127609449"} 2023-02-11 15:00:09,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:09,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:09,451 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:09,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 15:00:09,451 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:09,452 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:09,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:09,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:09,452 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127609451"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127609451"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127609451"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127609451"}]},"ts":"1676127609451"} 2023-02-11 15:00:09,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:09,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:09,453 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:09,453 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:09,453 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94a0d6907c18dd6eadf539b9aa0231e2 columnFamilyName cf 2023-02-11 15:00:09,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:09,454 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:09,454 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:09,454 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:09,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 15:00:09,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:09,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:09,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:09,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:09,455 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(310): Store=94a0d6907c18dd6eadf539b9aa0231e2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:09,456 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:09,456 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:09,456 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 704f07b3665f9baa8c5a5484e3bdbcd6 columnFamilyName cf 2023-02-11 15:00:09,457 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:09,457 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609456"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127609456"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127609456"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127609456"}]},"ts":"1676127609456"} 2023-02-11 15:00:09,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:09,458 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(310): Store=704f07b3665f9baa8c5a5484e3bdbcd6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:09,458 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:09,460 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:09,460 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:09,461 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=37 2023-02-11 15:00:09,461 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:09,461 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=37, state=SUCCESS; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785 in 183 msec 2023-02-11 15:00:09,461 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c301b6be53a2e3d2c7174a30417ca8e columnFamilyName cf 2023-02-11 15:00:09,463 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(310): Store=2c301b6be53a2e3d2c7174a30417ca8e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:09,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:09,464 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-02-11 15:00:09,464 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675 in 191 msec 2023-02-11 15:00:09,465 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN in 357 msec 2023-02-11 15:00:09,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:09,465 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=38 2023-02-11 15:00:09,466 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=38, state=SUCCESS; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743 in 191 msec 2023-02-11 15:00:09,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:09,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:09,467 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN in 360 msec 2023-02-11 15:00:09,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:09,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 94a0d6907c18dd6eadf539b9aa0231e2; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61294647, jitterRate=-0.08663858473300934}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:09,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 94a0d6907c18dd6eadf539b9aa0231e2: Running coprocessor pre-open hook at 1676127609450Writing region info on filesystem at 1676127609450Initializing all the Stores at 1676127609451 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127609451Cleaning up temporary data from old regions at 1676127609461 (+10 ms)Cleaning up detritus from prior splits at 1676127609461Running coprocessor post-open hooks at 1676127609468 (+7 ms)Region opened successfully at 1676127609469 (+1 ms) 2023-02-11 15:00:09,469 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN in 362 msec 2023-02-11 15:00:09,470 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2., pid=47, masterSystemTime=1676127609424 2023-02-11 15:00:09,472 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:09,472 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:09,472 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:09,473 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:09,473 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609473"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127609473"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127609473"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127609473"}]},"ts":"1676127609473"} 2023-02-11 15:00:09,474 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 704f07b3665f9baa8c5a5484e3bdbcd6; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59247038, jitterRate=-0.11715033650398254}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:09,474 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Running coprocessor pre-open hook at 1676127609452Writing region info on filesystem at 1676127609452Initializing all the Stores at 1676127609454 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127609454Cleaning up temporary data from old regions at 1676127609467 (+13 ms)Cleaning up detritus from prior splits at 1676127609468 (+1 ms)Running coprocessor post-open hooks at 1676127609474 (+6 ms)Region opened successfully at 1676127609474 2023-02-11 15:00:09,474 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:09,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6., pid=44, masterSystemTime=1676127609420 2023-02-11 15:00:09,476 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2c301b6be53a2e3d2c7174a30417ca8e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63842689, jitterRate=-0.04866980016231537}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:09,476 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2c301b6be53a2e3d2c7174a30417ca8e: Running coprocessor pre-open hook at 1676127609455Writing region info on filesystem at 1676127609455Initializing all the Stores at 1676127609458 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127609458Cleaning up temporary data from old regions at 1676127609469 (+11 ms)Cleaning up detritus from prior splits at 1676127609469Running coprocessor post-open hooks at 1676127609476 (+7 ms)Region opened successfully at 1676127609476 2023-02-11 15:00:09,477 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e., pid=48, masterSystemTime=1676127609422 2023-02-11 15:00:09,478 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:09,478 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:09,479 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:09,479 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609479"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127609479"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127609479"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127609479"}]},"ts":"1676127609479"} 2023-02-11 15:00:09,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:09,480 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:09,480 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=41 2023-02-11 15:00:09,480 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=41, state=SUCCESS; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785 in 205 msec 2023-02-11 15:00:09,480 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:09,480 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609480"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127609480"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127609480"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127609480"}]},"ts":"1676127609480"} 2023-02-11 15:00:09,483 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN in 376 msec 2023-02-11 15:00:09,484 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=39 2023-02-11 15:00:09,485 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=39, state=SUCCESS; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675 in 214 msec 2023-02-11 15:00:09,485 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=40 2023-02-11 15:00:09,485 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=40, state=SUCCESS; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743 in 208 msec 2023-02-11 15:00:09,487 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN in 381 msec 2023-02-11 15:00:09,488 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=36 2023-02-11 15:00:09,488 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN in 381 msec 2023-02-11 15:00:09,489 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127609489"}]},"ts":"1676127609489"} 2023-02-11 15:00:09,491 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLED in hbase:meta 2023-02-11 15:00:09,492 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676127607597' was successfully enabled. 2023-02-11 15:00:09,495 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676127607597 in 406 msec 2023-02-11 15:00:09,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 15:00:09,850 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 36 completed 2023-02-11 15:00:09,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:09,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:09,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:09,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:09,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:09,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:09,907 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676127607597 2023-02-11 15:00:09,907 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:09,909 INFO [Listener at localhost/39605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 15:00:09,929 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 15:00:09,932 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 15:00:09,935 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 15:00:09,938 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 15:00:09,940 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 15:00:09,950 INFO [Listener at localhost/39605] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:09,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676127607597 2023-02-11 15:00:09,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:09,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 15:00:09,955 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127609955"}]},"ts":"1676127609955"} 2023-02-11 15:00:09,957 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLING in hbase:meta 2023-02-11 15:00:09,959 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676127607597 to state=DISABLING 2023-02-11 15:00:09,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN}] 2023-02-11 15:00:09,962 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN 2023-02-11 15:00:09,962 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN 2023-02-11 15:00:09,962 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN 2023-02-11 15:00:09,963 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN 2023-02-11 15:00:09,963 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN 2023-02-11 15:00:09,963 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:09,964 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609963"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609963"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609963"}]},"ts":"1676127609963"} 2023-02-11 15:00:09,964 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:09,964 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:09,964 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127609964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609964"}]},"ts":"1676127609964"} 2023-02-11 15:00:09,964 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609964"}]},"ts":"1676127609964"} 2023-02-11 15:00:09,964 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:09,964 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:09,964 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609964"}]},"ts":"1676127609964"} 2023-02-11 15:00:09,965 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127609964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609964"}]},"ts":"1676127609964"} 2023-02-11 15:00:09,965 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=53, state=RUNNABLE; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:09,967 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=55, state=RUNNABLE; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:09,969 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=52, state=RUNNABLE; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:09,970 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=54, state=RUNNABLE; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:09,972 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=51, state=RUNNABLE; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:09,972 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN 2023-02-11 15:00:09,973 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:09,973 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127609973"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127609973"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127609973"}]},"ts":"1676127609973"} 2023-02-11 15:00:09,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=50, state=RUNNABLE; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:10,119 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:10,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing cd2163f692eb299f43cb55be4d4dc49f, disabling compactions & flushes 2023-02-11 15:00:10,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:10,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:10,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. after waiting 0 ms 2023-02-11 15:00:10,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:10,121 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing cd2163f692eb299f43cb55be4d4dc49f 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-11 15:00:10,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:10,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 78f87f7851c87dde38a4d94bc805db9b, disabling compactions & flushes 2023-02-11 15:00:10,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:10,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:10,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. after waiting 0 ms 2023-02-11 15:00:10,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:10,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 78f87f7851c87dde38a4d94bc805db9b 1/1 column families, dataSize=22.31 KB heapSize=48.20 KB 2023-02-11 15:00:10,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:10,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 94a0d6907c18dd6eadf539b9aa0231e2, disabling compactions & flushes 2023-02-11 15:00:10,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:10,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:10,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. after waiting 0 ms 2023-02-11 15:00:10,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:10,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 94a0d6907c18dd6eadf539b9aa0231e2 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-11 15:00:10,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 15:00:10,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c 2023-02-11 15:00:10,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba 2023-02-11 15:00:10,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad 2023-02-11 15:00:10,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/.tmp/cf/49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:10,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/.tmp/cf/2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:10,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/.tmp/cf/cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:10,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/.tmp/cf/2a35e2af579a4f78aad755c9c1a42e33 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:10,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/.tmp/cf/49a06ed7ffdc4dc9a2446e463bc2be02 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:10,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/.tmp/cf/cf109e42ce0a4e698dd14c8965b4c03a as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:10,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33, entries=28, sequenceid=8, filesize=9.7 K 2023-02-11 15:00:10,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for cd2163f692eb299f43cb55be4d4dc49f in 198ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:10,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676127607597' 2023-02-11 15:00:10,321 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02, entries=29, sequenceid=8, filesize=9.8 K 2023-02-11 15:00:10,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 94a0d6907c18dd6eadf539b9aa0231e2 in 199ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:10,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a, entries=341, sequenceid=8, filesize=63.0 K 2023-02-11 15:00:10,326 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.31 KB/22843, heapSize ~48.19 KB/49344, currentSize=0 B/0 for 78f87f7851c87dde38a4d94bc805db9b in 202ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:10,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:10,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:10,334 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:10,334 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:10,334 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:10,334 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 94a0d6907c18dd6eadf539b9aa0231e2: Waiting for close lock at 1676127610124Running coprocessor pre-close hooks at 1676127610124Disabling compacts and flushes for region at 1676127610124Disabling writes for close at 1676127610124Obtaining lock to block concurrent updates at 1676127610125 (+1 ms)Preparing flush snapshotting stores in 94a0d6907c18dd6eadf539b9aa0231e2 at 1676127610125Finished memstore snapshotting testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676127610131 (+6 ms)Flushing stores of testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. at 1676127610132 (+1 ms)Flushing 94a0d6907c18dd6eadf539b9aa0231e2/cf: creating writer at 1676127610134 (+2 ms)Flushing mob file 94a0d6907c18dd6eadf539b9aa0231e2/cf: appending metadata at 1676127610170 (+36 ms)Flushing mob file 94a0d6907c18dd6eadf539b9aa0231e2/cf: closing flushed file at 1676127610170Flushing 94a0d6907c18dd6eadf539b9aa0231e2/cf: appending metadata at 1676127610259 (+89 ms)Flushing 94a0d6907c18dd6eadf539b9aa0231e2/cf: closing flushed file at 1676127610262 (+3 ms)Flushing 94a0d6907c18dd6eadf539b9aa0231e2/cf: reopening flushed file at 1676127610312 (+50 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 94a0d6907c18dd6eadf539b9aa0231e2 in 199ms, sequenceid=8, compaction requested=false at 1676127610324 (+12 ms)Writing region close event to WAL at 1676127610329 (+5 ms)Running coprocessor post-close hooks at 1676127610334 (+5 ms)Closed at 1676127610334 2023-02-11 15:00:10,334 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 78f87f7851c87dde38a4d94bc805db9b: Waiting for close lock at 1676127610123Running coprocessor pre-close hooks at 1676127610123Disabling compacts and flushes for region at 1676127610123Disabling writes for close at 1676127610124 (+1 ms)Obtaining lock to block concurrent updates at 1676127610124Preparing flush snapshotting stores in 78f87f7851c87dde38a4d94bc805db9b at 1676127610124Finished memstore snapshotting testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b., syncing WAL and waiting on mvcc, flushsize=dataSize=22843, getHeapSize=49344, getOffHeapSize=0, getCellsCount=341 at 1676127610131 (+7 ms)Flushing stores of testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. at 1676127610132 (+1 ms)Flushing 78f87f7851c87dde38a4d94bc805db9b/cf: creating writer at 1676127610134 (+2 ms)Flushing mob file 78f87f7851c87dde38a4d94bc805db9b/cf: appending metadata at 1676127610183 (+49 ms)Flushing mob file 78f87f7851c87dde38a4d94bc805db9b/cf: closing flushed file at 1676127610184 (+1 ms)Flushing 78f87f7851c87dde38a4d94bc805db9b/cf: appending metadata at 1676127610259 (+75 ms)Flushing 78f87f7851c87dde38a4d94bc805db9b/cf: closing flushed file at 1676127610262 (+3 ms)Flushing 78f87f7851c87dde38a4d94bc805db9b/cf: reopening flushed file at 1676127610314 (+52 ms)Finished flush of dataSize ~22.31 KB/22843, heapSize ~48.19 KB/49344, currentSize=0 B/0 for 78f87f7851c87dde38a4d94bc805db9b in 202ms, sequenceid=8, compaction requested=false at 1676127610326 (+12 ms)Writing region close event to WAL at 1676127610329 (+3 ms)Running coprocessor post-close hooks at 1676127610334 (+5 ms)Closed at 1676127610334 2023-02-11 15:00:10,334 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:10,335 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for cd2163f692eb299f43cb55be4d4dc49f: Waiting for close lock at 1676127610119Running coprocessor pre-close hooks at 1676127610119Disabling compacts and flushes for region at 1676127610119Disabling writes for close at 1676127610120 (+1 ms)Obtaining lock to block concurrent updates at 1676127610121 (+1 ms)Preparing flush snapshotting stores in cd2163f692eb299f43cb55be4d4dc49f at 1676127610121Finished memstore snapshotting testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676127610131 (+10 ms)Flushing stores of testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. at 1676127610132 (+1 ms)Flushing cd2163f692eb299f43cb55be4d4dc49f/cf: creating writer at 1676127610134 (+2 ms)Flushing mob file cd2163f692eb299f43cb55be4d4dc49f/cf: appending metadata at 1676127610170 (+36 ms)Flushing mob file cd2163f692eb299f43cb55be4d4dc49f/cf: closing flushed file at 1676127610170Flushing cd2163f692eb299f43cb55be4d4dc49f/cf: appending metadata at 1676127610259 (+89 ms)Flushing cd2163f692eb299f43cb55be4d4dc49f/cf: closing flushed file at 1676127610262 (+3 ms)Flushing cd2163f692eb299f43cb55be4d4dc49f/cf: reopening flushed file at 1676127610308 (+46 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for cd2163f692eb299f43cb55be4d4dc49f in 198ms, sequenceid=8, compaction requested=false at 1676127610319 (+11 ms)Writing region close event to WAL at 1676127610328 (+9 ms)Running coprocessor post-close hooks at 1676127610334 (+6 ms)Closed at 1676127610334 2023-02-11 15:00:10,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:10,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:10,337 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7677bc32f6d3f7225f4cfdeb2068fc26, disabling compactions & flushes 2023-02-11 15:00:10,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:10,337 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:10,337 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. after waiting 0 ms 2023-02-11 15:00:10,337 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:10,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7677bc32f6d3f7225f4cfdeb2068fc26 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-11 15:00:10,337 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSED 2023-02-11 15:00:10,338 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127610337"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127610337"}]},"ts":"1676127610337"} 2023-02-11 15:00:10,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:10,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:10,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 704f07b3665f9baa8c5a5484e3bdbcd6, disabling compactions & flushes 2023-02-11 15:00:10,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:10,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:10,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. after waiting 0 ms 2023-02-11 15:00:10,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:10,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 704f07b3665f9baa8c5a5484e3bdbcd6 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-11 15:00:10,341 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSED 2023-02-11 15:00:10,341 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127610341"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127610341"}]},"ts":"1676127610341"} 2023-02-11 15:00:10,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:10,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:10,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2c301b6be53a2e3d2c7174a30417ca8e, disabling compactions & flushes 2023-02-11 15:00:10,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:10,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:10,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. after waiting 0 ms 2023-02-11 15:00:10,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:10,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 2c301b6be53a2e3d2c7174a30417ca8e 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-11 15:00:10,343 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSED 2023-02-11 15:00:10,344 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127610343"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127610343"}]},"ts":"1676127610343"} 2023-02-11 15:00:10,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=54 2023-02-11 15:00:10,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=54, state=SUCCESS; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785 in 373 msec 2023-02-11 15:00:10,350 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=55 2023-02-11 15:00:10,350 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=55, state=SUCCESS; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675 in 379 msec 2023-02-11 15:00:10,355 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=51 2023-02-11 15:00:10,355 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN in 388 msec 2023-02-11 15:00:10,355 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=51, state=SUCCESS; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743 in 375 msec 2023-02-11 15:00:10,356 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN in 390 msec 2023-02-11 15:00:10,358 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN in 396 msec 2023-02-11 15:00:10,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 2023-02-11 15:00:10,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 2023-02-11 15:00:10,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e 2023-02-11 15:00:10,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/.tmp/cf/830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:10,406 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/.tmp/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:10,406 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/.tmp/cf/ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:10,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/.tmp/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:10,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/.tmp/cf/830bf232363e40bbbca42f0cf5644c1d as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:10,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/.tmp/cf/ef1ac041d7184ad1af9b3666b7d994ae as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:10,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d, entries=34, sequenceid=8, filesize=10.7 K 2023-02-11 15:00:10,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae, entries=32, sequenceid=8, filesize=10.4 K 2023-02-11 15:00:10,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6, entries=36, sequenceid=8, filesize=11.0 K 2023-02-11 15:00:10,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 2c301b6be53a2e3d2c7174a30417ca8e in 84ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:10,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 7677bc32f6d3f7225f4cfdeb2068fc26 in 89ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:10,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 704f07b3665f9baa8c5a5484e3bdbcd6 in 85ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:10,435 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:10,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:10,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:10,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:10,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2c301b6be53a2e3d2c7174a30417ca8e: Waiting for close lock at 1676127610342Running coprocessor pre-close hooks at 1676127610342Disabling compacts and flushes for region at 1676127610342Disabling writes for close at 1676127610342Obtaining lock to block concurrent updates at 1676127610343 (+1 ms)Preparing flush snapshotting stores in 2c301b6be53a2e3d2c7174a30417ca8e at 1676127610343Finished memstore snapshotting testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676127610343Flushing stores of testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. at 1676127610345 (+2 ms)Flushing 2c301b6be53a2e3d2c7174a30417ca8e/cf: creating writer at 1676127610345Flushing mob file 2c301b6be53a2e3d2c7174a30417ca8e/cf: appending metadata at 1676127610352 (+7 ms)Flushing mob file 2c301b6be53a2e3d2c7174a30417ca8e/cf: closing flushed file at 1676127610352Flushing 2c301b6be53a2e3d2c7174a30417ca8e/cf: appending metadata at 1676127610389 (+37 ms)Flushing 2c301b6be53a2e3d2c7174a30417ca8e/cf: closing flushed file at 1676127610389Flushing 2c301b6be53a2e3d2c7174a30417ca8e/cf: reopening flushed file at 1676127610415 (+26 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 2c301b6be53a2e3d2c7174a30417ca8e in 84ms, sequenceid=8, compaction requested=false at 1676127610426 (+11 ms)Writing region close event to WAL at 1676127610428 (+2 ms)Running coprocessor post-close hooks at 1676127610436 (+8 ms)Closed at 1676127610436 2023-02-11 15:00:10,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:10,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:10,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Waiting for close lock at 1676127610340Running coprocessor pre-close hooks at 1676127610340Disabling compacts and flushes for region at 1676127610340Disabling writes for close at 1676127610341 (+1 ms)Obtaining lock to block concurrent updates at 1676127610341Preparing flush snapshotting stores in 704f07b3665f9baa8c5a5484e3bdbcd6 at 1676127610341Finished memstore snapshotting testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676127610341Flushing stores of testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. at 1676127610343 (+2 ms)Flushing 704f07b3665f9baa8c5a5484e3bdbcd6/cf: creating writer at 1676127610344 (+1 ms)Flushing mob file 704f07b3665f9baa8c5a5484e3bdbcd6/cf: appending metadata at 1676127610352 (+8 ms)Flushing mob file 704f07b3665f9baa8c5a5484e3bdbcd6/cf: closing flushed file at 1676127610352Flushing 704f07b3665f9baa8c5a5484e3bdbcd6/cf: appending metadata at 1676127610389 (+37 ms)Flushing 704f07b3665f9baa8c5a5484e3bdbcd6/cf: closing flushed file at 1676127610389Flushing 704f07b3665f9baa8c5a5484e3bdbcd6/cf: reopening flushed file at 1676127610416 (+27 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 704f07b3665f9baa8c5a5484e3bdbcd6 in 85ms, sequenceid=8, compaction requested=false at 1676127610428 (+12 ms)Writing region close event to WAL at 1676127610431 (+3 ms)Running coprocessor post-close hooks at 1676127610436 (+5 ms)Closed at 1676127610436 2023-02-11 15:00:10,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Waiting for close lock at 1676127610337Running coprocessor pre-close hooks at 1676127610337Disabling compacts and flushes for region at 1676127610337Disabling writes for close at 1676127610337Obtaining lock to block concurrent updates at 1676127610337Preparing flush snapshotting stores in 7677bc32f6d3f7225f4cfdeb2068fc26 at 1676127610337Finished memstore snapshotting testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676127610337Flushing stores of testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. at 1676127610338 (+1 ms)Flushing 7677bc32f6d3f7225f4cfdeb2068fc26/cf: creating writer at 1676127610339 (+1 ms)Flushing mob file 7677bc32f6d3f7225f4cfdeb2068fc26/cf: appending metadata at 1676127610352 (+13 ms)Flushing mob file 7677bc32f6d3f7225f4cfdeb2068fc26/cf: closing flushed file at 1676127610352Flushing 7677bc32f6d3f7225f4cfdeb2068fc26/cf: appending metadata at 1676127610391 (+39 ms)Flushing 7677bc32f6d3f7225f4cfdeb2068fc26/cf: closing flushed file at 1676127610391Flushing 7677bc32f6d3f7225f4cfdeb2068fc26/cf: reopening flushed file at 1676127610416 (+25 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 7677bc32f6d3f7225f4cfdeb2068fc26 in 89ms, sequenceid=8, compaction requested=false at 1676127610427 (+11 ms)Writing region close event to WAL at 1676127610431 (+4 ms)Running coprocessor post-close hooks at 1676127610436 (+5 ms)Closed at 1676127610436 2023-02-11 15:00:10,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:10,440 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSED 2023-02-11 15:00:10,440 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127610440"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127610440"}]},"ts":"1676127610440"} 2023-02-11 15:00:10,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:10,441 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSED 2023-02-11 15:00:10,441 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127610441"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127610441"}]},"ts":"1676127610441"} 2023-02-11 15:00:10,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:10,442 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSED 2023-02-11 15:00:10,442 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127610442"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127610442"}]},"ts":"1676127610442"} 2023-02-11 15:00:10,445 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=53 2023-02-11 15:00:10,445 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=53, state=SUCCESS; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743 in 477 msec 2023-02-11 15:00:10,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=50 2023-02-11 15:00:10,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=50, state=SUCCESS; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785 in 468 msec 2023-02-11 15:00:10,447 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN in 485 msec 2023-02-11 15:00:10,448 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=52 2023-02-11 15:00:10,448 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=52, state=SUCCESS; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675 in 476 msec 2023-02-11 15:00:10,448 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN in 486 msec 2023-02-11 15:00:10,450 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=49 2023-02-11 15:00:10,450 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN in 488 msec 2023-02-11 15:00:10,451 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127610451"}]},"ts":"1676127610451"} 2023-02-11 15:00:10,453 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLED in hbase:meta 2023-02-11 15:00:10,454 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676127607597 to state=DISABLED 2023-02-11 15:00:10,458 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676127607597 in 504 msec 2023-02-11 15:00:10,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 15:00:10,709 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 49 completed 2023-02-11 15:00:10,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:10,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676127610710 (current time:1676127610710). 2023-02-11 15:00:10,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 15:00:10,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(139): Connect 0x2a64838d to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:10,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60d4b936, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:10,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:10,721 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59244, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:10,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a64838d to 127.0.0.1:58940 2023-02-11 15:00:10,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:10,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 15:00:10,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:10,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:10,728 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:10,729 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE; timeout=600000, timestamp=1676128210729 2023-02-11 15:00:10,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:10,729 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676127607597 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 2023-02-11 15:00:10,729 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 15:00:10,729 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 15:00:10,746 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } 2023-02-11 15:00:10,749 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:10,749 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:10,750 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:10,750 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:10,751 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:10,751 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:10,751 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:10,751 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:10,751 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:10,751 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:10,751 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:10,751 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:10,751 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:10,752 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:10,753 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae] hfiles 2023-02-11 15:00:10,754 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:10,754 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a] hfiles 2023-02-11 15:00:10,754 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:10,755 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d] hfiles 2023-02-11 15:00:10,755 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:10,756 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33] hfiles 2023-02-11 15:00:10,756 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:10,756 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6] hfiles 2023-02-11 15:00:10,756 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:10,759 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156] hfiles 2023-02-11 15:00:10,759 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba 2023-02-11 15:00:10,759 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad 2023-02-11 15:00:10,759 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 2023-02-11 15:00:10,759 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e 2023-02-11 15:00:10,759 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c 2023-02-11 15:00:10,759 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 2023-02-11 15:00:10,763 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02] hfiles 2023-02-11 15:00:10,763 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:10,811 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } as finished. 2023-02-11 15:00:10,811 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676127607597 2023-02-11 15:00:10,812 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb0-1676127607597 2023-02-11 15:00:10,854 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb0-1676127607597 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/snaptb0-1676127607597 2023-02-11 15:00:10,880 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676127607597 of table testRestoreSnapshot-1676127607597 completed 2023-02-11 15:00:10,881 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb0-1676127607597 2023-02-11 15:00:10,881 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb0-1676127607597 2023-02-11 15:00:10,881 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676127607597 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 at 1676127610729Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } at 1676127610746 (+17 ms)Consolidate snapshot: snaptb0-1676127607597 at 1676127610811 (+65 ms)Loading Region manifests for snaptb0-1676127607597 at 1676127610811Writing data manifest for snaptb0-1676127607597 at 1676127610823 (+12 ms)Verifying snapshot: snaptb0-1676127607597 at 1676127610839 (+16 ms)Snapshot snaptb0-1676127607597 of table testRestoreSnapshot-1676127607597 completed at 1676127610880 (+41 ms) 2023-02-11 15:00:10,882 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:10,884 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE in 156 msec 2023-02-11 15:00:10,979 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 15:00:10,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } is done 2023-02-11 15:00:10,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 15:00:10,982 INFO [Listener at localhost/39605] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:10,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676127607597 2023-02-11 15:00:10,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:10,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 15:00:10,987 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676127607597 2023-02-11 15:00:10,987 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127610987"}]},"ts":"1676127610987"} 2023-02-11 15:00:10,988 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLING in hbase:meta 2023-02-11 15:00:10,993 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 15:00:10,994 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN}] 2023-02-11 15:00:10,996 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN 2023-02-11 15:00:10,996 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN 2023-02-11 15:00:10,996 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN 2023-02-11 15:00:10,996 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN 2023-02-11 15:00:10,997 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:10,997 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=true 2023-02-11 15:00:10,997 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:10,997 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:10,997 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN 2023-02-11 15:00:10,997 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=true 2023-02-11 15:00:10,997 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=true 2023-02-11 15:00:10,997 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:10,998 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=true 2023-02-11 15:00:10,998 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN 2023-02-11 15:00:10,998 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:10,998 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=true 2023-02-11 15:00:10,999 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:10,999 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=true 2023-02-11 15:00:11,147 INFO [jenkins-hbase4:38517] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 15:00:11,148 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:11,148 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:11,148 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127611147"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127611147"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127611147"}]},"ts":"1676127611147"} 2023-02-11 15:00:11,148 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:11,148 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:11,148 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:11,148 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127611148"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127611148"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127611148"}]},"ts":"1676127611148"} 2023-02-11 15:00:11,148 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127611147"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127611147"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127611147"}]},"ts":"1676127611147"} 2023-02-11 15:00:11,148 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127611148"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127611148"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127611148"}]},"ts":"1676127611148"} 2023-02-11 15:00:11,148 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127611148"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127611148"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127611148"}]},"ts":"1676127611148"} 2023-02-11 15:00:11,150 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=64, state=RUNNABLE; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:11,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=68, state=RUNNABLE; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:11,152 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=66, state=RUNNABLE; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:11,154 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=65, state=RUNNABLE; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:11,154 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=69, state=RUNNABLE; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:11,155 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:11,155 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127611155"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127611155"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127611155"}]},"ts":"1676127611155"} 2023-02-11 15:00:11,159 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=67, state=RUNNABLE; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:11,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 15:00:11,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:11,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 15:00:11,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:11,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:11,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 15:00:11,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:11,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:11,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:11,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:11,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:11,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:11,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:11,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:11,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 15:00:11,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:11,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:11,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:11,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:11,314 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:11,314 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:11,314 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:11,315 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:11,315 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:11,315 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:11,316 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:11,316 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:11,316 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:11,316 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cd2163f692eb299f43cb55be4d4dc49f columnFamilyName cf 2023-02-11 15:00:11,316 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 704f07b3665f9baa8c5a5484e3bdbcd6 columnFamilyName cf 2023-02-11 15:00:11,316 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7677bc32f6d3f7225f4cfdeb2068fc26 columnFamilyName cf 2023-02-11 15:00:11,329 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:11,329 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:11,330 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:11,330 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(310): Store=cd2163f692eb299f43cb55be4d4dc49f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:11,330 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(310): Store=704f07b3665f9baa8c5a5484e3bdbcd6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:11,330 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(310): Store=7677bc32f6d3f7225f4cfdeb2068fc26/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:11,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:11,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:11,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:11,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:11,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:11,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:11,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:11,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:11,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:11,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened cd2163f692eb299f43cb55be4d4dc49f; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60038635, jitterRate=-0.10535462200641632}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:11,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for cd2163f692eb299f43cb55be4d4dc49f: Running coprocessor pre-open hook at 1676127611311Writing region info on filesystem at 1676127611311Initializing all the Stores at 1676127611312 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127611312Cleaning up temporary data from old regions at 1676127611334 (+22 ms)Cleaning up detritus from prior splits at 1676127611335 (+1 ms)Running coprocessor post-open hooks at 1676127611339 (+4 ms)Region opened successfully at 1676127611339 2023-02-11 15:00:11,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 704f07b3665f9baa8c5a5484e3bdbcd6; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75433169, jitterRate=0.12404181063175201}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:11,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Running coprocessor pre-open hook at 1676127611309Writing region info on filesystem at 1676127611309Initializing all the Stores at 1676127611310 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127611310Cleaning up temporary data from old regions at 1676127611334 (+24 ms)Cleaning up detritus from prior splits at 1676127611335 (+1 ms)Running coprocessor post-open hooks at 1676127611340 (+5 ms)Region opened successfully at 1676127611340 2023-02-11 15:00:11,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7677bc32f6d3f7225f4cfdeb2068fc26; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68840486, jitterRate=0.025803178548812866}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:11,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Running coprocessor pre-open hook at 1676127611308Writing region info on filesystem at 1676127611308Initializing all the Stores at 1676127611310 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127611310Cleaning up temporary data from old regions at 1676127611334 (+24 ms)Cleaning up detritus from prior splits at 1676127611335 (+1 ms)Running coprocessor post-open hooks at 1676127611340 (+5 ms)Region opened successfully at 1676127611340 2023-02-11 15:00:11,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f., pid=73, masterSystemTime=1676127611306 2023-02-11 15:00:11,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6., pid=72, masterSystemTime=1676127611304 2023-02-11 15:00:11,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26., pid=70, masterSystemTime=1676127611303 2023-02-11 15:00:11,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:11,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:11,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:11,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 15:00:11,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:11,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:11,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:11,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:11,343 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:11,343 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127611343"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127611343"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127611343"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127611343"}]},"ts":"1676127611343"} 2023-02-11 15:00:11,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:11,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:11,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:11,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 15:00:11,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:11,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:11,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:11,345 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:11,345 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127611344"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127611344"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127611344"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127611344"}]},"ts":"1676127611344"} 2023-02-11 15:00:11,346 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:11,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:11,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:11,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:11,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 15:00:11,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:11,347 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:11,347 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127611347"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127611347"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127611347"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127611347"}]},"ts":"1676127611347"} 2023-02-11 15:00:11,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:11,348 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:11,348 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:11,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:11,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:11,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:11,348 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78f87f7851c87dde38a4d94bc805db9b columnFamilyName cf 2023-02-11 15:00:11,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=66 2023-02-11 15:00:11,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=66, state=SUCCESS; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675 in 193 msec 2023-02-11 15:00:11,351 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:11,351 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=65 2023-02-11 15:00:11,351 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:11,351 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=65, state=SUCCESS; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743 in 194 msec 2023-02-11 15:00:11,352 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN in 355 msec 2023-02-11 15:00:11,353 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:11,353 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:11,353 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:11,353 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:11,353 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=64 2023-02-11 15:00:11,353 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=64, state=SUCCESS; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785 in 199 msec 2023-02-11 15:00:11,353 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c301b6be53a2e3d2c7174a30417ca8e columnFamilyName cf 2023-02-11 15:00:11,353 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94a0d6907c18dd6eadf539b9aa0231e2 columnFamilyName cf 2023-02-11 15:00:11,355 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN in 358 msec 2023-02-11 15:00:11,356 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN in 359 msec 2023-02-11 15:00:11,359 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:11,359 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(310): Store=78f87f7851c87dde38a4d94bc805db9b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:11,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:11,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:11,362 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:11,362 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(310): Store=94a0d6907c18dd6eadf539b9aa0231e2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:11,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:11,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:11,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:11,367 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:11,367 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 78f87f7851c87dde38a4d94bc805db9b; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70691516, jitterRate=0.05338567495346069}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:11,367 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(310): Store=2c301b6be53a2e3d2c7174a30417ca8e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:11,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 78f87f7851c87dde38a4d94bc805db9b: Running coprocessor pre-open hook at 1676127611343Writing region info on filesystem at 1676127611343Initializing all the Stores at 1676127611344 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127611344Cleaning up temporary data from old regions at 1676127611362 (+18 ms)Cleaning up detritus from prior splits at 1676127611363 (+1 ms)Running coprocessor post-open hooks at 1676127611367 (+4 ms)Region opened successfully at 1676127611367 2023-02-11 15:00:11,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b., pid=74, masterSystemTime=1676127611304 2023-02-11 15:00:11,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:11,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:11,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:11,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:11,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:11,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 94a0d6907c18dd6eadf539b9aa0231e2; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66396702, jitterRate=-0.010612040758132935}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:11,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 94a0d6907c18dd6eadf539b9aa0231e2: Running coprocessor pre-open hook at 1676127611348Writing region info on filesystem at 1676127611348Initializing all the Stores at 1676127611351 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127611351Cleaning up temporary data from old regions at 1676127611366 (+15 ms)Cleaning up detritus from prior splits at 1676127611366Running coprocessor post-open hooks at 1676127611371 (+5 ms)Region opened successfully at 1676127611371 2023-02-11 15:00:11,371 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:11,371 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127611371"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127611371"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127611371"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127611371"}]},"ts":"1676127611371"} 2023-02-11 15:00:11,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2., pid=71, masterSystemTime=1676127611303 2023-02-11 15:00:11,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:11,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:11,375 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:11,376 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127611375"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127611375"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127611375"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127611375"}]},"ts":"1676127611375"} 2023-02-11 15:00:11,376 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=69 2023-02-11 15:00:11,376 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=69, state=SUCCESS; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675 in 220 msec 2023-02-11 15:00:11,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:11,378 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2c301b6be53a2e3d2c7174a30417ca8e; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68097849, jitterRate=0.014737024903297424}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:11,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2c301b6be53a2e3d2c7174a30417ca8e: Running coprocessor pre-open hook at 1676127611348Writing region info on filesystem at 1676127611348Initializing all the Stores at 1676127611350 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127611351 (+1 ms)Cleaning up temporary data from old regions at 1676127611371 (+20 ms)Cleaning up detritus from prior splits at 1676127611371Running coprocessor post-open hooks at 1676127611378 (+7 ms)Region opened successfully at 1676127611378 2023-02-11 15:00:11,378 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN in 382 msec 2023-02-11 15:00:11,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e., pid=75, masterSystemTime=1676127611306 2023-02-11 15:00:11,380 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=68 2023-02-11 15:00:11,381 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=68, state=SUCCESS; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785 in 227 msec 2023-02-11 15:00:11,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:11,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:11,382 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:11,382 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127611381"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127611381"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127611381"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127611381"}]},"ts":"1676127611381"} 2023-02-11 15:00:11,382 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN in 387 msec 2023-02-11 15:00:11,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=67 2023-02-11 15:00:11,386 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=67, state=SUCCESS; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743 in 226 msec 2023-02-11 15:00:11,388 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=67, resume processing ppid=63 2023-02-11 15:00:11,388 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN in 392 msec 2023-02-11 15:00:11,389 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127611389"}]},"ts":"1676127611389"} 2023-02-11 15:00:11,390 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLED in hbase:meta 2023-02-11 15:00:11,393 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676127607597' was successfully enabled. 2023-02-11 15:00:11,395 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676127607597 in 412 msec 2023-02-11 15:00:11,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 15:00:11,738 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 63 completed 2023-02-11 15:00:11,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:11,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:11,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:11,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:11,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:11,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:11,765 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676127607597 2023-02-11 15:00:11,765 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:11,766 INFO [Listener at localhost/39605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 15:00:12,030 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 15:00:12,030 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 15:00:12,095 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-11 15:00:12,096 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-11 15:00:12,179 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 15:00:12,343 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 15:00:12,509 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 15:00:12,642 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 15:00:13,642 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 15:00:13,642 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-11 15:00:14,060 DEBUG [Listener at localhost/39605] zookeeper.ReadOnlyZKClient(139): Connect 0x6d06cc97 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:14,068 DEBUG [Listener at localhost/39605] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@490facbf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:14,071 DEBUG [hconnection-0x290fa1b6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:14,074 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42422, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:14,078 DEBUG [hconnection-0x290fa1b6-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:14,081 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34108, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:14,193 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 15:00:14,197 DEBUG [hconnection-0x290fa1b6-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:14,199 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38684, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:14,296 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 15:00:14,419 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 15:00:14,555 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 15:00:14,655 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 15:00:15,774 INFO [Listener at localhost/39605] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:15,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676127607597 2023-02-11 15:00:15,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:15,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 15:00:15,780 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127615780"}]},"ts":"1676127615780"} 2023-02-11 15:00:15,782 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLING in hbase:meta 2023-02-11 15:00:15,784 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676127607597 to state=DISABLING 2023-02-11 15:00:15,785 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN}] 2023-02-11 15:00:15,787 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN 2023-02-11 15:00:15,788 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN 2023-02-11 15:00:15,788 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN 2023-02-11 15:00:15,788 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN 2023-02-11 15:00:15,789 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN 2023-02-11 15:00:15,789 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:15,789 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:15,789 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127615789"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127615789"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127615789"}]},"ts":"1676127615789"} 2023-02-11 15:00:15,789 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127615789"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127615789"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127615789"}]},"ts":"1676127615789"} 2023-02-11 15:00:15,789 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:15,790 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127615789"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127615789"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127615789"}]},"ts":"1676127615789"} 2023-02-11 15:00:15,790 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:15,790 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127615790"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127615790"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127615790"}]},"ts":"1676127615790"} 2023-02-11 15:00:15,790 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:15,791 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127615790"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127615790"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127615790"}]},"ts":"1676127615790"} 2023-02-11 15:00:15,791 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=79, state=RUNNABLE; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:15,792 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=80, state=RUNNABLE; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:15,793 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=78, state=RUNNABLE; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:15,794 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=77, state=RUNNABLE; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:15,795 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=82, state=RUNNABLE; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:15,799 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN 2023-02-11 15:00:15,800 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:15,801 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127615800"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127615800"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127615800"}]},"ts":"1676127615800"} 2023-02-11 15:00:15,802 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=81, state=RUNNABLE; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:15,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:15,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 704f07b3665f9baa8c5a5484e3bdbcd6, disabling compactions & flushes 2023-02-11 15:00:15,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:15,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:15,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. after waiting 0 ms 2023-02-11 15:00:15,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:15,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 704f07b3665f9baa8c5a5484e3bdbcd6 1/1 column families, dataSize=2.75 KB heapSize=6.16 KB 2023-02-11 15:00:15,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:15,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2c301b6be53a2e3d2c7174a30417ca8e, disabling compactions & flushes 2023-02-11 15:00:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7677bc32f6d3f7225f4cfdeb2068fc26, disabling compactions & flushes 2023-02-11 15:00:15,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:15,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. after waiting 0 ms 2023-02-11 15:00:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. after waiting 0 ms 2023-02-11 15:00:15,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 2c301b6be53a2e3d2c7174a30417ca8e 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-11 15:00:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:15,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7677bc32f6d3f7225f4cfdeb2068fc26 1/1 column families, dataSize=1.57 KB heapSize=3.63 KB 2023-02-11 15:00:15,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 2023-02-11 15:00:15,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 2023-02-11 15:00:15,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 2023-02-11 15:00:15,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/.tmp/cf/ea92c7cd9a554d4a849955a308ed7e13 2023-02-11 15:00:15,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/.tmp/cf/80b2a4abbc984f83a8efd755415eaf73 2023-02-11 15:00:15,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/.tmp/cf/ea92c7cd9a554d4a849955a308ed7e13 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/ea92c7cd9a554d4a849955a308ed7e13 2023-02-11 15:00:15,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/.tmp/cf/a11a6bb05820487da1f0bf1a178b54e0 2023-02-11 15:00:16,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/.tmp/cf/80b2a4abbc984f83a8efd755415eaf73 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/80b2a4abbc984f83a8efd755415eaf73 2023-02-11 15:00:16,001 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/ea92c7cd9a554d4a849955a308ed7e13, entries=29, sequenceid=15, filesize=9.8 K 2023-02-11 15:00:16,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/.tmp/cf/a11a6bb05820487da1f0bf1a178b54e0 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/a11a6bb05820487da1f0bf1a178b54e0 2023-02-11 15:00:16,003 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 2c301b6be53a2e3d2c7174a30417ca8e in 54ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:16,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/80b2a4abbc984f83a8efd755415eaf73, entries=24, sequenceid=15, filesize=9.0 K 2023-02-11 15:00:16,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/a11a6bb05820487da1f0bf1a178b54e0, entries=42, sequenceid=15, filesize=12.0 K 2023-02-11 15:00:16,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 7677bc32f6d3f7225f4cfdeb2068fc26 in 62ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:16,011 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:16,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.75 KB/2812, heapSize ~6.14 KB/6288, currentSize=0 B/0 for 704f07b3665f9baa8c5a5484e3bdbcd6 in 64ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:16,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:16,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2c301b6be53a2e3d2c7174a30417ca8e: Waiting for close lock at 1676127615948Running coprocessor pre-close hooks at 1676127615948Disabling compacts and flushes for region at 1676127615948Disabling writes for close at 1676127615949 (+1 ms)Obtaining lock to block concurrent updates at 1676127615949Preparing flush snapshotting stores in 2c301b6be53a2e3d2c7174a30417ca8e at 1676127615949Finished memstore snapshotting testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676127615950 (+1 ms)Flushing stores of testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. at 1676127615952 (+2 ms)Flushing 2c301b6be53a2e3d2c7174a30417ca8e/cf: creating writer at 1676127615952Flushing mob file 2c301b6be53a2e3d2c7174a30417ca8e/cf: appending metadata at 1676127615960 (+8 ms)Flushing mob file 2c301b6be53a2e3d2c7174a30417ca8e/cf: closing flushed file at 1676127615960Flushing 2c301b6be53a2e3d2c7174a30417ca8e/cf: appending metadata at 1676127615983 (+23 ms)Flushing 2c301b6be53a2e3d2c7174a30417ca8e/cf: closing flushed file at 1676127615983Flushing 2c301b6be53a2e3d2c7174a30417ca8e/cf: reopening flushed file at 1676127615997 (+14 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 2c301b6be53a2e3d2c7174a30417ca8e in 54ms, sequenceid=15, compaction requested=false at 1676127616003 (+6 ms)Writing region close event to WAL at 1676127616007 (+4 ms)Running coprocessor post-close hooks at 1676127616014 (+7 ms)Closed at 1676127616015 (+1 ms) 2023-02-11 15:00:16,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:16,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:16,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing cd2163f692eb299f43cb55be4d4dc49f, disabling compactions & flushes 2023-02-11 15:00:16,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:16,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:16,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. after waiting 0 ms 2023-02-11 15:00:16,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:16,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing cd2163f692eb299f43cb55be4d4dc49f 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-11 15:00:16,020 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSED 2023-02-11 15:00:16,021 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127616020"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127616020"}]},"ts":"1676127616020"} 2023-02-11 15:00:16,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:16,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:16,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Waiting for close lock at 1676127615949Running coprocessor pre-close hooks at 1676127615949Disabling compacts and flushes for region at 1676127615949Disabling writes for close at 1676127615949Obtaining lock to block concurrent updates at 1676127615949Preparing flush snapshotting stores in 7677bc32f6d3f7225f4cfdeb2068fc26 at 1676127615949Finished memstore snapshotting testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26., syncing WAL and waiting on mvcc, flushsize=dataSize=1606, getHeapSize=3696, getOffHeapSize=0, getCellsCount=24 at 1676127615950 (+1 ms)Flushing stores of testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. at 1676127615951 (+1 ms)Flushing 7677bc32f6d3f7225f4cfdeb2068fc26/cf: creating writer at 1676127615952 (+1 ms)Flushing mob file 7677bc32f6d3f7225f4cfdeb2068fc26/cf: appending metadata at 1676127615960 (+8 ms)Flushing mob file 7677bc32f6d3f7225f4cfdeb2068fc26/cf: closing flushed file at 1676127615960Flushing 7677bc32f6d3f7225f4cfdeb2068fc26/cf: appending metadata at 1676127615983 (+23 ms)Flushing 7677bc32f6d3f7225f4cfdeb2068fc26/cf: closing flushed file at 1676127615983Flushing 7677bc32f6d3f7225f4cfdeb2068fc26/cf: reopening flushed file at 1676127616001 (+18 ms)Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 7677bc32f6d3f7225f4cfdeb2068fc26 in 62ms, sequenceid=15, compaction requested=false at 1676127616011 (+10 ms)Writing region close event to WAL at 1676127616015 (+4 ms)Running coprocessor post-close hooks at 1676127616023 (+8 ms)Closed at 1676127616023 2023-02-11 15:00:16,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:16,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:16,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Waiting for close lock at 1676127615946Running coprocessor pre-close hooks at 1676127615946Disabling compacts and flushes for region at 1676127615946Disabling writes for close at 1676127615947 (+1 ms)Obtaining lock to block concurrent updates at 1676127615948 (+1 ms)Preparing flush snapshotting stores in 704f07b3665f9baa8c5a5484e3bdbcd6 at 1676127615948Finished memstore snapshotting testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6., syncing WAL and waiting on mvcc, flushsize=dataSize=2812, getHeapSize=6288, getOffHeapSize=0, getCellsCount=42 at 1676127615949 (+1 ms)Flushing stores of testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. at 1676127615950 (+1 ms)Flushing 704f07b3665f9baa8c5a5484e3bdbcd6/cf: creating writer at 1676127615951 (+1 ms)Flushing mob file 704f07b3665f9baa8c5a5484e3bdbcd6/cf: appending metadata at 1676127615960 (+9 ms)Flushing mob file 704f07b3665f9baa8c5a5484e3bdbcd6/cf: closing flushed file at 1676127615960Flushing 704f07b3665f9baa8c5a5484e3bdbcd6/cf: appending metadata at 1676127615984 (+24 ms)Flushing 704f07b3665f9baa8c5a5484e3bdbcd6/cf: closing flushed file at 1676127615984Flushing 704f07b3665f9baa8c5a5484e3bdbcd6/cf: reopening flushed file at 1676127616004 (+20 ms)Finished flush of dataSize ~2.75 KB/2812, heapSize ~6.14 KB/6288, currentSize=0 B/0 for 704f07b3665f9baa8c5a5484e3bdbcd6 in 64ms, sequenceid=15, compaction requested=false at 1676127616011 (+7 ms)Writing region close event to WAL at 1676127616019 (+8 ms)Running coprocessor post-close hooks at 1676127616026 (+7 ms)Closed at 1676127616026 2023-02-11 15:00:16,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:16,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:16,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 94a0d6907c18dd6eadf539b9aa0231e2, disabling compactions & flushes 2023-02-11 15:00:16,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:16,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:16,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. after waiting 0 ms 2023-02-11 15:00:16,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:16,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 94a0d6907c18dd6eadf539b9aa0231e2 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-11 15:00:16,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=80 2023-02-11 15:00:16,027 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSED 2023-02-11 15:00:16,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=80, state=SUCCESS; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,33165,1676127604743 in 231 msec 2023-02-11 15:00:16,027 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127616027"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127616027"}]},"ts":"1676127616027"} 2023-02-11 15:00:16,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:16,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:16,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 78f87f7851c87dde38a4d94bc805db9b, disabling compactions & flushes 2023-02-11 15:00:16,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:16,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:16,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. after waiting 0 ms 2023-02-11 15:00:16,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:16,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 78f87f7851c87dde38a4d94bc805db9b 1/1 column families, dataSize=23.03 KB heapSize=49.75 KB 2023-02-11 15:00:16,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 15:00:16,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN in 242 msec 2023-02-11 15:00:16,032 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSED 2023-02-11 15:00:16,032 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127616032"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127616032"}]},"ts":"1676127616032"} 2023-02-11 15:00:16,035 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=77 2023-02-11 15:00:16,035 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=77, state=SUCCESS; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785 in 236 msec 2023-02-11 15:00:16,040 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=79 2023-02-11 15:00:16,040 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=79, state=SUCCESS; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675 in 243 msec 2023-02-11 15:00:16,041 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN in 250 msec 2023-02-11 15:00:16,042 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN in 255 msec 2023-02-11 15:00:16,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 2023-02-11 15:00:16,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc 2023-02-11 15:00:16,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 2023-02-11 15:00:16,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/.tmp/cf/08c52f6c5115434fa8273a8c2162f5e1 2023-02-11 15:00:16,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/.tmp/cf/4fefa1144af6408195c155495f67763b 2023-02-11 15:00:16,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=23.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/.tmp/cf/46fbcfe2640b44ad951ffc998805ea49 2023-02-11 15:00:16,080 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/.tmp/cf/08c52f6c5115434fa8273a8c2162f5e1 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/08c52f6c5115434fa8273a8c2162f5e1 2023-02-11 15:00:16,080 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/.tmp/cf/4fefa1144af6408195c155495f67763b as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/4fefa1144af6408195c155495f67763b 2023-02-11 15:00:16,081 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/.tmp/cf/46fbcfe2640b44ad951ffc998805ea49 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/46fbcfe2640b44ad951ffc998805ea49 2023-02-11 15:00:16,084 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/4fefa1144af6408195c155495f67763b, entries=26, sequenceid=15, filesize=9.3 K 2023-02-11 15:00:16,084 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/08c52f6c5115434fa8273a8c2162f5e1, entries=27, sequenceid=15, filesize=9.5 K 2023-02-11 15:00:16,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/46fbcfe2640b44ad951ffc998805ea49, entries=352, sequenceid=15, filesize=64.9 K 2023-02-11 15:00:16,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for cd2163f692eb299f43cb55be4d4dc49f in 65ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:16,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 94a0d6907c18dd6eadf539b9aa0231e2 in 58ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:16,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.03 KB/23580, heapSize ~49.73 KB/50928, currentSize=0 B/0 for 78f87f7851c87dde38a4d94bc805db9b in 56ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:16,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:16,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:16,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for cd2163f692eb299f43cb55be4d4dc49f: Waiting for close lock at 1676127616019Running coprocessor pre-close hooks at 1676127616019Disabling compacts and flushes for region at 1676127616019Disabling writes for close at 1676127616020 (+1 ms)Obtaining lock to block concurrent updates at 1676127616020Preparing flush snapshotting stores in cd2163f692eb299f43cb55be4d4dc49f at 1676127616020Finished memstore snapshotting testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676127616020Flushing stores of testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. at 1676127616021 (+1 ms)Flushing cd2163f692eb299f43cb55be4d4dc49f/cf: creating writer at 1676127616021Flushing mob file cd2163f692eb299f43cb55be4d4dc49f/cf: appending metadata at 1676127616028 (+7 ms)Flushing mob file cd2163f692eb299f43cb55be4d4dc49f/cf: closing flushed file at 1676127616028Flushing cd2163f692eb299f43cb55be4d4dc49f/cf: appending metadata at 1676127616066 (+38 ms)Flushing cd2163f692eb299f43cb55be4d4dc49f/cf: closing flushed file at 1676127616066Flushing cd2163f692eb299f43cb55be4d4dc49f/cf: reopening flushed file at 1676127616081 (+15 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for cd2163f692eb299f43cb55be4d4dc49f in 65ms, sequenceid=15, compaction requested=false at 1676127616085 (+4 ms)Writing region close event to WAL at 1676127616089 (+4 ms)Running coprocessor post-close hooks at 1676127616097 (+8 ms)Closed at 1676127616098 (+1 ms) 2023-02-11 15:00:16,101 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:16,101 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:16,101 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSED 2023-02-11 15:00:16,102 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127616101"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127616101"}]},"ts":"1676127616101"} 2023-02-11 15:00:16,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:16,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 78f87f7851c87dde38a4d94bc805db9b: Waiting for close lock at 1676127616030Running coprocessor pre-close hooks at 1676127616030Disabling compacts and flushes for region at 1676127616030Disabling writes for close at 1676127616030Obtaining lock to block concurrent updates at 1676127616030Preparing flush snapshotting stores in 78f87f7851c87dde38a4d94bc805db9b at 1676127616030Finished memstore snapshotting testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b., syncing WAL and waiting on mvcc, flushsize=dataSize=23580, getHeapSize=50928, getOffHeapSize=0, getCellsCount=352 at 1676127616031 (+1 ms)Flushing stores of testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. at 1676127616032 (+1 ms)Flushing 78f87f7851c87dde38a4d94bc805db9b/cf: creating writer at 1676127616033 (+1 ms)Flushing mob file 78f87f7851c87dde38a4d94bc805db9b/cf: appending metadata at 1676127616042 (+9 ms)Flushing mob file 78f87f7851c87dde38a4d94bc805db9b/cf: closing flushed file at 1676127616042Flushing 78f87f7851c87dde38a4d94bc805db9b/cf: appending metadata at 1676127616068 (+26 ms)Flushing 78f87f7851c87dde38a4d94bc805db9b/cf: closing flushed file at 1676127616068Flushing 78f87f7851c87dde38a4d94bc805db9b/cf: reopening flushed file at 1676127616082 (+14 ms)Finished flush of dataSize ~23.03 KB/23580, heapSize ~49.73 KB/50928, currentSize=0 B/0 for 78f87f7851c87dde38a4d94bc805db9b in 56ms, sequenceid=15, compaction requested=false at 1676127616089 (+7 ms)Writing region close event to WAL at 1676127616094 (+5 ms)Running coprocessor post-close hooks at 1676127616102 (+8 ms)Closed at 1676127616102 2023-02-11 15:00:16,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:16,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:16,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 94a0d6907c18dd6eadf539b9aa0231e2: Waiting for close lock at 1676127616027Running coprocessor pre-close hooks at 1676127616027Disabling compacts and flushes for region at 1676127616027Disabling writes for close at 1676127616027Obtaining lock to block concurrent updates at 1676127616027Preparing flush snapshotting stores in 94a0d6907c18dd6eadf539b9aa0231e2 at 1676127616027Finished memstore snapshotting testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676127616027Flushing stores of testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. at 1676127616028 (+1 ms)Flushing 94a0d6907c18dd6eadf539b9aa0231e2/cf: creating writer at 1676127616029 (+1 ms)Flushing mob file 94a0d6907c18dd6eadf539b9aa0231e2/cf: appending metadata at 1676127616039 (+10 ms)Flushing mob file 94a0d6907c18dd6eadf539b9aa0231e2/cf: closing flushed file at 1676127616039Flushing 94a0d6907c18dd6eadf539b9aa0231e2/cf: appending metadata at 1676127616066 (+27 ms)Flushing 94a0d6907c18dd6eadf539b9aa0231e2/cf: closing flushed file at 1676127616066Flushing 94a0d6907c18dd6eadf539b9aa0231e2/cf: reopening flushed file at 1676127616081 (+15 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 94a0d6907c18dd6eadf539b9aa0231e2 in 58ms, sequenceid=15, compaction requested=false at 1676127616087 (+6 ms)Writing region close event to WAL at 1676127616094 (+7 ms)Running coprocessor post-close hooks at 1676127616104 (+10 ms)Closed at 1676127616104 2023-02-11 15:00:16,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:16,106 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSED 2023-02-11 15:00:16,106 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127616106"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127616106"}]},"ts":"1676127616106"} 2023-02-11 15:00:16,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:16,107 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSED 2023-02-11 15:00:16,107 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127616107"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127616107"}]},"ts":"1676127616107"} 2023-02-11 15:00:16,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=78 2023-02-11 15:00:16,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=78, state=SUCCESS; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743 in 312 msec 2023-02-11 15:00:16,110 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN in 323 msec 2023-02-11 15:00:16,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=82 2023-02-11 15:00:16,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=82, state=SUCCESS; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675 in 313 msec 2023-02-11 15:00:16,111 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=81 2023-02-11 15:00:16,111 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=81, state=SUCCESS; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785 in 307 msec 2023-02-11 15:00:16,112 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN in 326 msec 2023-02-11 15:00:16,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=81, resume processing ppid=76 2023-02-11 15:00:16,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN in 326 msec 2023-02-11 15:00:16,114 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127616114"}]},"ts":"1676127616114"} 2023-02-11 15:00:16,115 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLED in hbase:meta 2023-02-11 15:00:16,117 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676127607597 to state=DISABLED 2023-02-11 15:00:16,119 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676127607597 in 342 msec 2023-02-11 15:00:16,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 15:00:16,533 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 76 completed 2023-02-11 15:00:16,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:16,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676127616534 (current time:1676127616534). 2023-02-11 15:00:16,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 15:00:16,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(139): Connect 0x13f10391 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:16,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d078289, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:16,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:16,547 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42432, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:16,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13f10391 to 127.0.0.1:58940 2023-02-11 15:00:16,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:16,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 15:00:16,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:16,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:16,552 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:16,552 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE; timeout=600000, timestamp=1676128216552 2023-02-11 15:00:16,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:16,552 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676127607597 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 2023-02-11 15:00:16,553 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 15:00:16,553 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 15:00:16,568 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } 2023-02-11 15:00:16,573 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:16,573 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:16,574 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:16,574 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:16,574 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:16,574 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:16,576 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:16,576 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:16,576 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:16,576 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:16,576 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:16,576 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:16,578 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:16,578 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:16,584 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/4fefa1144af6408195c155495f67763b] hfiles 2023-02-11 15:00:16,584 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:16,584 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/4fefa1144af6408195c155495f67763b 2023-02-11 15:00:16,585 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/46fbcfe2640b44ad951ffc998805ea49, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a] hfiles 2023-02-11 15:00:16,585 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/46fbcfe2640b44ad951ffc998805ea49 2023-02-11 15:00:16,585 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:16,586 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/ea92c7cd9a554d4a849955a308ed7e13] hfiles 2023-02-11 15:00:16,586 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:16,586 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/08c52f6c5115434fa8273a8c2162f5e1, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02] hfiles 2023-02-11 15:00:16,586 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/ea92c7cd9a554d4a849955a308ed7e13 2023-02-11 15:00:16,586 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/08c52f6c5115434fa8273a8c2162f5e1 2023-02-11 15:00:16,586 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:16,587 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/80b2a4abbc984f83a8efd755415eaf73, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae] hfiles 2023-02-11 15:00:16,587 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/80b2a4abbc984f83a8efd755415eaf73 2023-02-11 15:00:16,587 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:16,588 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/a11a6bb05820487da1f0bf1a178b54e0] hfiles 2023-02-11 15:00:16,588 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:16,588 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/a11a6bb05820487da1f0bf1a178b54e0 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156] hfiles 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 2023-02-11 15:00:16,597 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 2023-02-11 15:00:16,803 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 15:00:16,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } is done 2023-02-11 15:00:16,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }' is still in progress! 2023-02-11 15:00:16,805 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-11 15:00:17,059 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } as finished. 2023-02-11 15:00:17,059 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676127607597 2023-02-11 15:00:17,060 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb1-1676127607597 2023-02-11 15:00:17,094 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb1-1676127607597 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/snaptb1-1676127607597 2023-02-11 15:00:17,114 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676127607597 of table testRestoreSnapshot-1676127607597 completed 2023-02-11 15:00:17,114 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb1-1676127607597 2023-02-11 15:00:17,114 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb1-1676127607597 2023-02-11 15:00:17,114 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676127607597 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 at 1676127616552Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } at 1676127616568 (+16 ms)Consolidate snapshot: snaptb1-1676127607597 at 1676127617059 (+491 ms)Loading Region manifests for snaptb1-1676127607597 at 1676127617060 (+1 ms)Writing data manifest for snaptb1-1676127607597 at 1676127617065 (+5 ms)Verifying snapshot: snaptb1-1676127607597 at 1676127617078 (+13 ms)Snapshot snaptb1-1676127607597 of table testRestoreSnapshot-1676127607597 completed at 1676127617114 (+36 ms) 2023-02-11 15:00:17,116 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:17,117 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE in 566 msec 2023-02-11 15:00:17,305 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 15:00:17,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } is done 2023-02-11 15:00:17,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 15:00:17,323 INFO [Listener at localhost/39605] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676127607597-1676127617323 2023-02-11 15:00:17,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676127607597-1676127617323 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:17,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676127617325 (current time:1676127617325). 2023-02-11 15:00:17,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 15:00:17,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(139): Connect 0x0ee7528b to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:17,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73adcd8a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:17,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:17,333 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42442, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:17,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ee7528b to 127.0.0.1:58940 2023-02-11 15:00:17,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:17,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 15:00:17,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676127607597-1676127617323 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:17,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:17,337 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:17,338 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE; timeout=600000, timestamp=1676128217338 2023-02-11 15:00:17,338 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676127607597-1676127617323 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 2023-02-11 15:00:17,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676127607597-1676127617323 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:17,339 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676127607597-1676127617323 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 15:00:17,339 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 15:00:17,349 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676127607597-1676127617323 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } 2023-02-11 15:00:17,350 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:17,350 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:17,350 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:17,350 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:17,350 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:17,350 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:17,351 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:17,351 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:17,352 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:17,352 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:17,352 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:17,352 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:17,353 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:17,353 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:17,356 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/80b2a4abbc984f83a8efd755415eaf73, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae] hfiles 2023-02-11 15:00:17,356 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/08c52f6c5115434fa8273a8c2162f5e1, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02] hfiles 2023-02-11 15:00:17,356 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/80b2a4abbc984f83a8efd755415eaf73 2023-02-11 15:00:17,356 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:17,356 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/08c52f6c5115434fa8273a8c2162f5e1 2023-02-11 15:00:17,356 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/4fefa1144af6408195c155495f67763b] hfiles 2023-02-11 15:00:17,356 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:17,356 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:17,357 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/ea92c7cd9a554d4a849955a308ed7e13] hfiles 2023-02-11 15:00:17,357 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:17,357 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/ea92c7cd9a554d4a849955a308ed7e13 2023-02-11 15:00:17,356 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/46fbcfe2640b44ad951ffc998805ea49, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a] hfiles 2023-02-11 15:00:17,357 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/4fefa1144af6408195c155495f67763b 2023-02-11 15:00:17,357 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/46fbcfe2640b44ad951ffc998805ea49 2023-02-11 15:00:17,357 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:17,358 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/a11a6bb05820487da1f0bf1a178b54e0] hfiles 2023-02-11 15:00:17,358 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:17,358 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/a11a6bb05820487da1f0bf1a178b54e0 2023-02-11 15:00:17,392 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6d06cc97 to 127.0.0.1:58940 2023-02-11 15:00:17,392 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156] hfiles 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 2023-02-11 15:00:17,395 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 2023-02-11 15:00:17,426 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676127607597-1676127617323 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } as finished. 2023-02-11 15:00:17,426 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676127607597-1676127617323 2023-02-11 15:00:17,427 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676127607597-1676127617323 2023-02-11 15:00:17,459 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676127607597-1676127617323 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/hbase-failsafe-snaptb0-1676127607597-1676127617323 2023-02-11 15:00:17,483 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676127607597-1676127617323 of table testRestoreSnapshot-1676127607597 completed 2023-02-11 15:00:17,483 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676127607597-1676127617323 2023-02-11 15:00:17,484 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676127607597-1676127617323 2023-02-11 15:00:17,484 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676127607597-1676127617323 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 at 1676127617338Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676127607597-1676127617323 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } at 1676127617349 (+11 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676127607597-1676127617323 at 1676127617426 (+77 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676127607597-1676127617323 at 1676127617426Writing data manifest for hbase-failsafe-snaptb0-1676127607597-1676127617323 at 1676127617430 (+4 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676127607597-1676127617323 at 1676127617445 (+15 ms)Snapshot hbase-failsafe-snaptb0-1676127607597-1676127617323 of table testRestoreSnapshot-1676127607597 completed at 1676127617483 (+38 ms) 2023-02-11 15:00:17,485 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:17,487 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE in 149 msec 2023-02-11 15:00:17,589 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 15:00:17,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676127607597-1676127617323 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } is done 2023-02-11 15:00:17,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676127607597-1676127617323 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 15:00:17,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676127607597 snapshot=name: "snaptb0-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127610710 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 15:00:17,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676127607597 as table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:17,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-11 15:00:17,642 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000002 2023-02-11 15:00:17,643 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000001 2023-02-11 15:00:17,643 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000002 2023-02-11 15:00:17,644 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } 2023-02-11 15:00:17,648 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127610710 type: DISABLED version: 2 ttl: 0 2023-02-11 15:00:17,649 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597 2023-02-11 15:00:17,656 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:17,662 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 2023-02-11 15:00:17,664 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 2023-02-11 15:00:17,666 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 2023-02-11 15:00:17,667 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 2023-02-11 15:00:17,669 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 2023-02-11 15:00:17,670 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc 2023-02-11 15:00:17,670 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:17,670 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:17,670 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:17,671 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:17,671 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:17,671 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:17,681 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/a11a6bb05820487da1f0bf1a178b54e0 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/a11a6bb05820487da1f0bf1a178b54e0 2023-02-11 15:00:17,681 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/08c52f6c5115434fa8273a8c2162f5e1 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/08c52f6c5115434fa8273a8c2162f5e1 2023-02-11 15:00:17,682 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/46fbcfe2640b44ad951ffc998805ea49 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/46fbcfe2640b44ad951ffc998805ea49 2023-02-11 15:00:17,682 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/ea92c7cd9a554d4a849955a308ed7e13 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/ea92c7cd9a554d4a849955a308ed7e13 2023-02-11 15:00:17,682 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/80b2a4abbc984f83a8efd755415eaf73 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/80b2a4abbc984f83a8efd755415eaf73 2023-02-11 15:00:17,683 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/4fefa1144af6408195c155495f67763b to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/4fefa1144af6408195c155495f67763b 2023-02-11 15:00:17,683 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127610710 type: DISABLED version: 2 ttl: 0 2023-02-11 15:00:17,685 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127617685"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:17,685 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127617685"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:17,685 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127617685"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:17,685 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127617685"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:17,685 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127617685"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:17,685 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127617685"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:17,688 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 15:00:17,688 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 15:00:17,688 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127617686"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127617686"}]},"ts":"1676127617686"} 2023-02-11 15:00:17,688 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127617686"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127617686"}]},"ts":"1676127617686"} 2023-02-11 15:00:17,688 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127617686"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127617686"}]},"ts":"1676127617686"} 2023-02-11 15:00:17,689 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127617686"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127617686"}]},"ts":"1676127617686"} 2023-02-11 15:00:17,689 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127617686"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127617686"}]},"ts":"1676127617686"} 2023-02-11 15:00:17,689 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127617686"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127617686"}]},"ts":"1676127617686"} 2023-02-11 15:00:17,691 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 15:00:17,691 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 15:00:17,691 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 15:00:17,692 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676127607597 completed! 2023-02-11 15:00:17,694 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676127607597 snapshot=name: "snaptb0-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127610710 type: DISABLED version: 2 ttl: 0 ) in 83 msec 2023-02-11 15:00:17,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-11 15:00:17,877 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676127607597, procId: 91 completed 2023-02-11 15:00:17,877 INFO [Listener at localhost/39605] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676127607597-1676127617323 2023-02-11 15:00:17,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1676127607597-1676127617323" 2023-02-11 15:00:17,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676127607597-1676127617323 2023-02-11 15:00:17,885 INFO [Listener at localhost/39605] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:17,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676127607597 2023-02-11 15:00:17,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:17,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 15:00:17,889 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676127607597 2023-02-11 15:00:17,889 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127617889"}]},"ts":"1676127617889"} 2023-02-11 15:00:17,891 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLING in hbase:meta 2023-02-11 15:00:17,893 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 15:00:17,895 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN}] 2023-02-11 15:00:17,897 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN 2023-02-11 15:00:17,897 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN 2023-02-11 15:00:17,897 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN 2023-02-11 15:00:17,897 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN 2023-02-11 15:00:17,898 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:17,898 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:17,898 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:17,899 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:17,899 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN 2023-02-11 15:00:17,899 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN 2023-02-11 15:00:17,900 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:17,900 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:18,049 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 15:00:18,049 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 15:00:18,049 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 15:00:18,049 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 15:00:18,049 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 15:00:18,052 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:18,052 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:18,052 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127618052"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127618052"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127618052"}]},"ts":"1676127618052"} 2023-02-11 15:00:18,052 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:18,052 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:18,053 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127618052"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127618052"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127618052"}]},"ts":"1676127618052"} 2023-02-11 15:00:18,053 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127618052"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127618052"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127618052"}]},"ts":"1676127618052"} 2023-02-11 15:00:18,052 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:18,052 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127618052"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127618052"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127618052"}]},"ts":"1676127618052"} 2023-02-11 15:00:18,053 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127618052"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127618052"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127618052"}]},"ts":"1676127618052"} 2023-02-11 15:00:18,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=94, state=RUNNABLE; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:18,055 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=97, state=RUNNABLE; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:18,056 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=93, state=RUNNABLE; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:18,057 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=98, state=RUNNABLE; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:18,058 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=96, state=RUNNABLE; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:18,059 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:18,059 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127618059"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127618059"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127618059"}]},"ts":"1676127618059"} 2023-02-11 15:00:18,061 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=95, state=RUNNABLE; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:18,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 15:00:18,209 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:18,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 15:00:18,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:18,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:18,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:18,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:18,211 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:18,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:18,212 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:18,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 15:00:18,212 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:18,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:18,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:18,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:18,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:18,212 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cd2163f692eb299f43cb55be4d4dc49f columnFamilyName cf 2023-02-11 15:00:18,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:18,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 15:00:18,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:18,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:18,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:18,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:18,214 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:18,214 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:18,214 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:18,214 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:18,215 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7677bc32f6d3f7225f4cfdeb2068fc26 columnFamilyName cf 2023-02-11 15:00:18,215 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:18,215 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:18,215 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78f87f7851c87dde38a4d94bc805db9b columnFamilyName cf 2023-02-11 15:00:18,217 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:18,218 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(310): Store=cd2163f692eb299f43cb55be4d4dc49f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:18,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:18,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:18,221 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:18,222 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(310): Store=7677bc32f6d3f7225f4cfdeb2068fc26/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:18,222 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:18,222 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(310): Store=78f87f7851c87dde38a4d94bc805db9b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:18,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:18,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:18,224 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:18,224 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:18,224 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:18,224 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened cd2163f692eb299f43cb55be4d4dc49f; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60998069, jitterRate=-0.09105794131755829}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:18,224 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for cd2163f692eb299f43cb55be4d4dc49f: Running coprocessor pre-open hook at 1676127618210Writing region info on filesystem at 1676127618210Initializing all the Stores at 1676127618210Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127618210Cleaning up temporary data from old regions at 1676127618221 (+11 ms)Cleaning up detritus from prior splits at 1676127618221Running coprocessor post-open hooks at 1676127618224 (+3 ms)Region opened successfully at 1676127618224 2023-02-11 15:00:18,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f., pid=99, masterSystemTime=1676127618206 2023-02-11 15:00:18,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:18,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:18,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:18,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:18,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:18,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 15:00:18,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7677bc32f6d3f7225f4cfdeb2068fc26; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65819251, jitterRate=-0.019216731190681458}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:18,227 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:18,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:18,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Running coprocessor pre-open hook at 1676127618212Writing region info on filesystem at 1676127618212Initializing all the Stores at 1676127618213 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127618213Cleaning up temporary data from old regions at 1676127618224 (+11 ms)Cleaning up detritus from prior splits at 1676127618225 (+1 ms)Running coprocessor post-open hooks at 1676127618227 (+2 ms)Region opened successfully at 1676127618227 2023-02-11 15:00:18,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:18,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 78f87f7851c87dde38a4d94bc805db9b; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61299756, jitterRate=-0.08656245470046997}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:18,228 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127618227"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127618227"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127618227"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127618227"}]},"ts":"1676127618227"} 2023-02-11 15:00:18,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:18,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:18,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 78f87f7851c87dde38a4d94bc805db9b: Running coprocessor pre-open hook at 1676127618213Writing region info on filesystem at 1676127618213Initializing all the Stores at 1676127618214 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127618214Cleaning up temporary data from old regions at 1676127618225 (+11 ms)Cleaning up detritus from prior splits at 1676127618225Running coprocessor post-open hooks at 1676127618228 (+3 ms)Region opened successfully at 1676127618228 2023-02-11 15:00:18,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26., pid=101, masterSystemTime=1676127618209 2023-02-11 15:00:18,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b., pid=102, masterSystemTime=1676127618211 2023-02-11 15:00:18,229 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:18,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:18,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:18,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:18,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 15:00:18,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:18,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:18,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:18,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:18,231 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:18,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:18,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:18,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:18,231 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127618231"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127618231"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127618231"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127618231"}]},"ts":"1676127618231"} 2023-02-11 15:00:18,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 15:00:18,231 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:18,231 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:18,232 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=94 2023-02-11 15:00:18,232 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:18,232 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=94, state=SUCCESS; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,42505,1676127604785 in 176 msec 2023-02-11 15:00:18,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:18,232 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127618231"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127618231"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127618231"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127618231"}]},"ts":"1676127618231"} 2023-02-11 15:00:18,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:18,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:18,232 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94a0d6907c18dd6eadf539b9aa0231e2 columnFamilyName cf 2023-02-11 15:00:18,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:18,232 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:18,233 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:18,233 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:18,233 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:18,234 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN in 339 msec 2023-02-11 15:00:18,234 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c301b6be53a2e3d2c7174a30417ca8e columnFamilyName cf 2023-02-11 15:00:18,234 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:18,235 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:18,235 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 704f07b3665f9baa8c5a5484e3bdbcd6 columnFamilyName cf 2023-02-11 15:00:18,236 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=93 2023-02-11 15:00:18,236 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=93, state=SUCCESS; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,41207,1676127604675 in 177 msec 2023-02-11 15:00:18,237 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=98 2023-02-11 15:00:18,237 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=98, state=SUCCESS; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743 in 178 msec 2023-02-11 15:00:18,238 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:18,238 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(310): Store=94a0d6907c18dd6eadf539b9aa0231e2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:18,239 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:18,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:18,240 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:18,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:18,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:18,243 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 94a0d6907c18dd6eadf539b9aa0231e2; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71587155, jitterRate=0.06673173606395721}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:18,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 94a0d6907c18dd6eadf539b9aa0231e2: Running coprocessor pre-open hook at 1676127618228Writing region info on filesystem at 1676127618228Initializing all the Stores at 1676127618229 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127618229Cleaning up temporary data from old regions at 1676127618240 (+11 ms)Cleaning up detritus from prior splits at 1676127618241 (+1 ms)Running coprocessor post-open hooks at 1676127618243 (+2 ms)Region opened successfully at 1676127618243 2023-02-11 15:00:18,243 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2., pid=100, masterSystemTime=1676127618206 2023-02-11 15:00:18,244 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(310): Store=704f07b3665f9baa8c5a5484e3bdbcd6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:18,244 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN in 343 msec 2023-02-11 15:00:18,244 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(310): Store=2c301b6be53a2e3d2c7174a30417ca8e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:18,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:18,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:18,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:18,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:18,246 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN in 342 msec 2023-02-11 15:00:18,246 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:18,247 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127618246"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127618246"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127618246"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127618246"}]},"ts":"1676127618246"} 2023-02-11 15:00:18,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:18,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:18,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:18,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:18,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 704f07b3665f9baa8c5a5484e3bdbcd6; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64522146, jitterRate=-0.03854510188102722}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:18,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Running coprocessor pre-open hook at 1676127618232Writing region info on filesystem at 1676127618232Initializing all the Stores at 1676127618233 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127618233Cleaning up temporary data from old regions at 1676127618247 (+14 ms)Cleaning up detritus from prior splits at 1676127618248 (+1 ms)Running coprocessor post-open hooks at 1676127618251 (+3 ms)Region opened successfully at 1676127618251 2023-02-11 15:00:18,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2c301b6be53a2e3d2c7174a30417ca8e; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59808274, jitterRate=-0.10878726840019226}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:18,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2c301b6be53a2e3d2c7174a30417ca8e: Running coprocessor pre-open hook at 1676127618231Writing region info on filesystem at 1676127618231Initializing all the Stores at 1676127618232 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127618232Cleaning up temporary data from old regions at 1676127618248 (+16 ms)Cleaning up detritus from prior splits at 1676127618249 (+1 ms)Running coprocessor post-open hooks at 1676127618252 (+3 ms)Region opened successfully at 1676127618252 2023-02-11 15:00:18,252 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=97 2023-02-11 15:00:18,252 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=97, state=SUCCESS; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785 in 195 msec 2023-02-11 15:00:18,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6., pid=104, masterSystemTime=1676127618211 2023-02-11 15:00:18,253 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e., pid=103, masterSystemTime=1676127618209 2023-02-11 15:00:18,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN in 357 msec 2023-02-11 15:00:18,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:18,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:18,254 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:18,254 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127618254"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127618254"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127618254"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127618254"}]},"ts":"1676127618254"} 2023-02-11 15:00:18,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:18,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:18,255 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:18,255 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127618255"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127618255"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127618255"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127618255"}]},"ts":"1676127618255"} 2023-02-11 15:00:18,258 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=95 2023-02-11 15:00:18,258 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=95, state=SUCCESS; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743 in 195 msec 2023-02-11 15:00:18,258 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=96 2023-02-11 15:00:18,258 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=96, state=SUCCESS; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,41207,1676127604675 in 199 msec 2023-02-11 15:00:18,259 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN in 363 msec 2023-02-11 15:00:18,260 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=96, resume processing ppid=92 2023-02-11 15:00:18,260 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN in 363 msec 2023-02-11 15:00:18,261 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127618261"}]},"ts":"1676127618261"} 2023-02-11 15:00:18,262 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLED in hbase:meta 2023-02-11 15:00:18,263 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676127607597' was successfully enabled. 2023-02-11 15:00:18,265 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676127607597 in 378 msec 2023-02-11 15:00:18,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 15:00:18,641 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 92 completed 2023-02-11 15:00:18,641 DEBUG [Listener at localhost/39605] zookeeper.ReadOnlyZKClient(139): Connect 0x272618c8 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:18,646 DEBUG [Listener at localhost/39605] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@251148a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:18,648 DEBUG [hconnection-0x7c1707f4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:18,650 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42448, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:18,746 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 15:00:18,749 DEBUG [hconnection-0x7c1707f4-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:18,751 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34114, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:18,834 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 15:00:18,837 DEBUG [hconnection-0x7c1707f4-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:18,840 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38688, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:18,940 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 15:00:19,058 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 15:00:19,145 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 15:00:20,064 INFO [Listener at localhost/39605] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-11 15:00:20,069 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127620068"}]},"ts":"1676127620068"} 2023-02-11 15:00:20,070 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLING in hbase:meta 2023-02-11 15:00:20,072 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676127607597 to state=DISABLING 2023-02-11 15:00:20,073 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN}] 2023-02-11 15:00:20,076 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN 2023-02-11 15:00:20,076 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN 2023-02-11 15:00:20,076 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN 2023-02-11 15:00:20,076 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN 2023-02-11 15:00:20,077 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN 2023-02-11 15:00:20,077 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:20,077 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:20,077 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127620077"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127620077"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127620077"}]},"ts":"1676127620077"} 2023-02-11 15:00:20,078 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620077"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127620077"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127620077"}]},"ts":"1676127620077"} 2023-02-11 15:00:20,077 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:20,078 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620077"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127620077"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127620077"}]},"ts":"1676127620077"} 2023-02-11 15:00:20,078 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:20,078 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620078"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127620078"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127620078"}]},"ts":"1676127620078"} 2023-02-11 15:00:20,078 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:20,079 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620078"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127620078"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127620078"}]},"ts":"1676127620078"} 2023-02-11 15:00:20,079 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=111, state=RUNNABLE; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:20,082 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=108, state=RUNNABLE; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:20,084 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=110, state=RUNNABLE; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:20,085 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=109, state=RUNNABLE; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:20,086 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN 2023-02-11 15:00:20,086 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=107, state=RUNNABLE; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:20,087 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:20,087 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127620087"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127620087"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127620087"}]},"ts":"1676127620087"} 2023-02-11 15:00:20,089 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=106, state=RUNNABLE; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:20,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:20,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 78f87f7851c87dde38a4d94bc805db9b, disabling compactions & flushes 2023-02-11 15:00:20,235 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:20,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:20,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. after waiting 0 ms 2023-02-11 15:00:20,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:20,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:20,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:20,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2c301b6be53a2e3d2c7174a30417ca8e, disabling compactions & flushes 2023-02-11 15:00:20,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 94a0d6907c18dd6eadf539b9aa0231e2, disabling compactions & flushes 2023-02-11 15:00:20,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:20,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:20,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:20,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:20,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. after waiting 0 ms 2023-02-11 15:00:20,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. after waiting 0 ms 2023-02-11 15:00:20,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:20,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:20,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 15:00:20,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:20,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 15:00:20,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 78f87f7851c87dde38a4d94bc805db9b: Waiting for close lock at 1676127620234Running coprocessor pre-close hooks at 1676127620234Disabling compacts and flushes for region at 1676127620234Disabling writes for close at 1676127620235 (+1 ms)Writing region close event to WAL at 1676127620238 (+3 ms)Running coprocessor post-close hooks at 1676127620248 (+10 ms)Closed at 1676127620248 2023-02-11 15:00:20,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 15:00:20,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:20,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2c301b6be53a2e3d2c7174a30417ca8e: Waiting for close lock at 1676127620238Running coprocessor pre-close hooks at 1676127620238Disabling compacts and flushes for region at 1676127620238Disabling writes for close at 1676127620240 (+2 ms)Writing region close event to WAL at 1676127620245 (+5 ms)Running coprocessor post-close hooks at 1676127620249 (+4 ms)Closed at 1676127620249 2023-02-11 15:00:20,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:20,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 94a0d6907c18dd6eadf539b9aa0231e2: Waiting for close lock at 1676127620238Running coprocessor pre-close hooks at 1676127620238Disabling compacts and flushes for region at 1676127620238Disabling writes for close at 1676127620240 (+2 ms)Writing region close event to WAL at 1676127620245 (+5 ms)Running coprocessor post-close hooks at 1676127620249 (+4 ms)Closed at 1676127620249 2023-02-11 15:00:20,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:20,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:20,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 704f07b3665f9baa8c5a5484e3bdbcd6, disabling compactions & flushes 2023-02-11 15:00:20,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:20,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:20,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. after waiting 0 ms 2023-02-11 15:00:20,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:20,251 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSED 2023-02-11 15:00:20,251 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127620251"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620251"}]},"ts":"1676127620251"} 2023-02-11 15:00:20,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:20,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:20,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7677bc32f6d3f7225f4cfdeb2068fc26, disabling compactions & flushes 2023-02-11 15:00:20,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:20,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:20,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. after waiting 0 ms 2023-02-11 15:00:20,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:20,252 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSED 2023-02-11 15:00:20,252 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620252"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620252"}]},"ts":"1676127620252"} 2023-02-11 15:00:20,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:20,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:20,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing cd2163f692eb299f43cb55be4d4dc49f, disabling compactions & flushes 2023-02-11 15:00:20,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:20,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:20,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. after waiting 0 ms 2023-02-11 15:00:20,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:20,253 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSED 2023-02-11 15:00:20,254 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620253"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620253"}]},"ts":"1676127620253"} 2023-02-11 15:00:20,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 15:00:20,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:20,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Waiting for close lock at 1676127620250Running coprocessor pre-close hooks at 1676127620250Disabling compacts and flushes for region at 1676127620250Disabling writes for close at 1676127620250Writing region close event to WAL at 1676127620251 (+1 ms)Running coprocessor post-close hooks at 1676127620256 (+5 ms)Closed at 1676127620256 2023-02-11 15:00:20,256 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=111 2023-02-11 15:00:20,256 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=111, state=SUCCESS; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743 in 173 msec 2023-02-11 15:00:20,256 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=109 2023-02-11 15:00:20,256 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=109, state=SUCCESS; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,41207,1676127604675 in 170 msec 2023-02-11 15:00:20,257 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=110 2023-02-11 15:00:20,257 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=110, state=SUCCESS; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785 in 171 msec 2023-02-11 15:00:20,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:20,258 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN in 183 msec 2023-02-11 15:00:20,258 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN in 183 msec 2023-02-11 15:00:20,258 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSED 2023-02-11 15:00:20,259 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620258"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620258"}]},"ts":"1676127620258"} 2023-02-11 15:00:20,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 15:00:20,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 15:00:20,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:20,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Waiting for close lock at 1676127620252Running coprocessor pre-close hooks at 1676127620252Disabling compacts and flushes for region at 1676127620252Disabling writes for close at 1676127620252Writing region close event to WAL at 1676127620253 (+1 ms)Running coprocessor post-close hooks at 1676127620259 (+6 ms)Closed at 1676127620259 2023-02-11 15:00:20,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:20,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for cd2163f692eb299f43cb55be4d4dc49f: Waiting for close lock at 1676127620253Running coprocessor pre-close hooks at 1676127620253Disabling compacts and flushes for region at 1676127620253Disabling writes for close at 1676127620253Writing region close event to WAL at 1676127620254 (+1 ms)Running coprocessor post-close hooks at 1676127620259 (+5 ms)Closed at 1676127620259 2023-02-11 15:00:20,259 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN in 184 msec 2023-02-11 15:00:20,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:20,261 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSED 2023-02-11 15:00:20,261 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127620261"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620261"}]},"ts":"1676127620261"} 2023-02-11 15:00:20,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:20,262 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSED 2023-02-11 15:00:20,262 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620261"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620261"}]},"ts":"1676127620261"} 2023-02-11 15:00:20,262 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=108 2023-02-11 15:00:20,262 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=108, state=SUCCESS; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743 in 180 msec 2023-02-11 15:00:20,263 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN in 189 msec 2023-02-11 15:00:20,265 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=106 2023-02-11 15:00:20,265 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=106, state=SUCCESS; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,41207,1676127604675 in 176 msec 2023-02-11 15:00:20,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=107 2023-02-11 15:00:20,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=107, state=SUCCESS; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,42505,1676127604785 in 178 msec 2023-02-11 15:00:20,266 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN in 192 msec 2023-02-11 15:00:20,267 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=107, resume processing ppid=105 2023-02-11 15:00:20,267 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN in 193 msec 2023-02-11 15:00:20,268 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127620268"}]},"ts":"1676127620268"} 2023-02-11 15:00:20,269 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLED in hbase:meta 2023-02-11 15:00:20,271 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676127607597 to state=DISABLED 2023-02-11 15:00:20,273 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676127607597 in 208 msec 2023-02-11 15:00:20,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-11 15:00:20,320 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 105 completed 2023-02-11 15:00:20,328 INFO [Listener at localhost/39605] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676127607597-1676127620328 2023-02-11 15:00:20,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676127607597-1676127620328 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:20,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676127620329 (current time:1676127620329). 2023-02-11 15:00:20,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 15:00:20,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(139): Connect 0x5f361ca9 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:20,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@452c0955, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:20,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:20,338 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42464, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:20,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f361ca9 to 127.0.0.1:58940 2023-02-11 15:00:20,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:20,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 15:00:20,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676127607597-1676127620328 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:20,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:20,342 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:20,343 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE; timeout=600000, timestamp=1676128220343 2023-02-11 15:00:20,343 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676127607597-1676127620328 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676127607597-1676127620328 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:20,343 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676127607597-1676127620328 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 15:00:20,343 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 15:00:20,352 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676127607597-1676127620328 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } 2023-02-11 15:00:20,353 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:20,353 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:20,353 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:20,353 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:20,354 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:20,354 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:20,354 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:20,354 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:20,354 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:20,354 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:20,354 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:20,354 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:20,354 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:20,354 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:20,356 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33] hfiles 2023-02-11 15:00:20,356 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:20,356 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae] hfiles 2023-02-11 15:00:20,356 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:20,358 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a] hfiles 2023-02-11 15:00:20,358 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:20,359 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d] hfiles 2023-02-11 15:00:20,359 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:20,359 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6] hfiles 2023-02-11 15:00:20,359 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02] hfiles 2023-02-11 15:00:20,359 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:20,359 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:20,361 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156] hfiles 2023-02-11 15:00:20,362 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba 2023-02-11 15:00:20,362 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad 2023-02-11 15:00:20,362 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 2023-02-11 15:00:20,362 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e 2023-02-11 15:00:20,362 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c 2023-02-11 15:00:20,362 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 2023-02-11 15:00:20,399 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676127607597-1676127620328 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } as finished. 2023-02-11 15:00:20,399 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676127607597-1676127620328 2023-02-11 15:00:20,400 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676127607597-1676127620328 2023-02-11 15:00:20,420 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676127607597-1676127620328 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676127607597-1676127620328 2023-02-11 15:00:20,439 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676127607597-1676127620328 of table testRestoreSnapshot-1676127607597 completed 2023-02-11 15:00:20,439 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676127607597-1676127620328 2023-02-11 15:00:20,439 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676127607597-1676127620328 2023-02-11 15:00:20,439 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676127607597-1676127620328 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 at 1676127620343Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676127607597-1676127620328 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } at 1676127620352 (+9 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676127607597-1676127620328 at 1676127620399 (+47 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676127607597-1676127620328 at 1676127620400 (+1 ms)Writing data manifest for hbase-failsafe-emptySnaptb-1676127607597-1676127620328 at 1676127620403 (+3 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676127607597-1676127620328 at 1676127620412 (+9 ms)Snapshot hbase-failsafe-emptySnaptb-1676127607597-1676127620328 of table testRestoreSnapshot-1676127607597 completed at 1676127620439 (+27 ms) 2023-02-11 15:00:20,441 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:20,442 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE in 100 msec 2023-02-11 15:00:20,593 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 15:00:20,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676127607597-1676127620328 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } is done 2023-02-11 15:00:20,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676127607597-1676127620328 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 15:00:20,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676127607597 snapshot=name: "emptySnaptb-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127608775 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 15:00:20,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676127607597 as table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-11 15:00:20,611 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000003 2023-02-11 15:00:20,612 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000002 2023-02-11 15:00:20,612 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000003 2023-02-11 15:00:20,613 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } 2023-02-11 15:00:20,614 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127608775 type: DISABLED version: 2 ttl: 0 2023-02-11 15:00:20,614 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,621 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,624 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba 2023-02-11 15:00:20,625 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad 2023-02-11 15:00:20,626 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 2023-02-11 15:00:20,627 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e 2023-02-11 15:00:20,628 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c 2023-02-11 15:00:20,629 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 2023-02-11 15:00:20,629 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:20,629 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:20,629 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:20,629 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:20,629 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:20,629 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:20,634 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:20,634 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:20,634 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:20,634 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:20,635 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:20,635 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:20,635 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127608775 type: DISABLED version: 2 ttl: 0 2023-02-11 15:00:20,637 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127620636"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:20,637 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127620636"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:20,637 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127620636"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:20,637 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127620636"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:20,637 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127620636"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:20,637 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127620636"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:20,639 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 15:00:20,639 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 15:00:20,639 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620637"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620637"}]},"ts":"1676127620637"} 2023-02-11 15:00:20,639 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620637"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620637"}]},"ts":"1676127620637"} 2023-02-11 15:00:20,640 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127620637"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620637"}]},"ts":"1676127620637"} 2023-02-11 15:00:20,640 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127620637"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620637"}]},"ts":"1676127620637"} 2023-02-11 15:00:20,640 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620637"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620637"}]},"ts":"1676127620637"} 2023-02-11 15:00:20,640 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127620637"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127620637"}]},"ts":"1676127620637"} 2023-02-11 15:00:20,642 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 15:00:20,642 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 15:00:20,642 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 15:00:20,642 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676127607597 completed! 2023-02-11 15:00:20,644 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676127607597 snapshot=name: "emptySnaptb-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127608775 type: DISABLED version: 2 ttl: 0 ) in 44 msec 2023-02-11 15:00:20,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-11 15:00:20,853 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676127607597, procId: 119 completed 2023-02-11 15:00:20,853 INFO [Listener at localhost/39605] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676127607597-1676127620328 2023-02-11 15:00:20,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1676127607597-1676127620328" 2023-02-11 15:00:20,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676127607597-1676127620328 2023-02-11 15:00:20,857 INFO [Listener at localhost/39605] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 15:00:20,861 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676127607597 2023-02-11 15:00:20,861 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127620861"}]},"ts":"1676127620861"} 2023-02-11 15:00:20,862 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLING in hbase:meta 2023-02-11 15:00:20,865 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 15:00:20,865 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN}] 2023-02-11 15:00:20,867 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN 2023-02-11 15:00:20,867 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN 2023-02-11 15:00:20,867 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN 2023-02-11 15:00:20,867 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN 2023-02-11 15:00:20,869 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN 2023-02-11 15:00:20,869 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:20,869 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:20,869 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:20,869 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:20,870 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:20,870 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN 2023-02-11 15:00:20,870 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:21,019 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 15:00:21,019 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 15:00:21,019 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 15:00:21,019 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 15:00:21,019 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 15:00:21,023 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:21,023 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:21,023 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:21,023 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:21,024 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127621023"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621023"}]},"ts":"1676127621023"} 2023-02-11 15:00:21,023 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:21,024 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621023"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621023"}]},"ts":"1676127621023"} 2023-02-11 15:00:21,023 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127621023"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621023"}]},"ts":"1676127621023"} 2023-02-11 15:00:21,023 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621023"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621023"}]},"ts":"1676127621023"} 2023-02-11 15:00:21,024 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621023"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621023"}]},"ts":"1676127621023"} 2023-02-11 15:00:21,025 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=121, state=RUNNABLE; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:21,026 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=122, state=RUNNABLE; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:21,027 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=126, state=RUNNABLE; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:21,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=125, state=RUNNABLE; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:21,028 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:21,029 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621028"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621028"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621028"}]},"ts":"1676127621028"} 2023-02-11 15:00:21,029 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=124, state=RUNNABLE; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:21,031 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=123, state=RUNNABLE; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:21,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 15:00:21,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:21,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:21,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 15:00:21,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 15:00:21,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:21,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 15:00:21,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:21,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:21,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:21,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:21,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:21,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:21,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:21,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:21,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:21,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:21,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:21,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:21,182 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:21,182 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:21,182 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:21,183 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:21,183 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:21,183 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:21,183 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:21,183 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:21,183 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:21,184 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c301b6be53a2e3d2c7174a30417ca8e columnFamilyName cf 2023-02-11 15:00:21,184 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 704f07b3665f9baa8c5a5484e3bdbcd6 columnFamilyName cf 2023-02-11 15:00:21,184 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94a0d6907c18dd6eadf539b9aa0231e2 columnFamilyName cf 2023-02-11 15:00:21,185 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(310): Store=2c301b6be53a2e3d2c7174a30417ca8e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:21,185 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(310): Store=704f07b3665f9baa8c5a5484e3bdbcd6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:21,185 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(310): Store=94a0d6907c18dd6eadf539b9aa0231e2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:21,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:21,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:21,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:21,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:21,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:21,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:21,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:21,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:21,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:21,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 704f07b3665f9baa8c5a5484e3bdbcd6; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61855433, jitterRate=-0.07828222215175629}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:21,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Running coprocessor pre-open hook at 1676127621181Writing region info on filesystem at 1676127621181Initializing all the Stores at 1676127621181Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127621182 (+1 ms)Cleaning up temporary data from old regions at 1676127621187 (+5 ms)Cleaning up detritus from prior splits at 1676127621187Running coprocessor post-open hooks at 1676127621190 (+3 ms)Region opened successfully at 1676127621190 2023-02-11 15:00:21,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 94a0d6907c18dd6eadf539b9aa0231e2; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64140333, jitterRate=-0.04423455893993378}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:21,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2c301b6be53a2e3d2c7174a30417ca8e; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62950038, jitterRate=-0.06197133660316467}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:21,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2c301b6be53a2e3d2c7174a30417ca8e: Running coprocessor pre-open hook at 1676127621181Writing region info on filesystem at 1676127621181Initializing all the Stores at 1676127621181Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127621181Cleaning up temporary data from old regions at 1676127621187 (+6 ms)Cleaning up detritus from prior splits at 1676127621187Running coprocessor post-open hooks at 1676127621190 (+3 ms)Region opened successfully at 1676127621190 2023-02-11 15:00:21,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 94a0d6907c18dd6eadf539b9aa0231e2: Running coprocessor pre-open hook at 1676127621181Writing region info on filesystem at 1676127621181Initializing all the Stores at 1676127621181Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127621181Cleaning up temporary data from old regions at 1676127621187 (+6 ms)Cleaning up detritus from prior splits at 1676127621187Running coprocessor post-open hooks at 1676127621190 (+3 ms)Region opened successfully at 1676127621190 2023-02-11 15:00:21,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6., pid=132, masterSystemTime=1676127621178 2023-02-11 15:00:21,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e., pid=131, masterSystemTime=1676127621177 2023-02-11 15:00:21,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2., pid=130, masterSystemTime=1676127621177 2023-02-11 15:00:21,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:21,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:21,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:21,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 15:00:21,192 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:21,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:21,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:21,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:21,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:21,192 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621192"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127621192"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127621192"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127621192"}]},"ts":"1676127621192"} 2023-02-11 15:00:21,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:21,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:21,193 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:21,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 15:00:21,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:21,193 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621193"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127621193"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127621193"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127621193"}]},"ts":"1676127621193"} 2023-02-11 15:00:21,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:21,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:21,193 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:21,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:21,193 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:21,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:21,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:21,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 15:00:21,193 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:21,194 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621193"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127621193"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127621193"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127621193"}]},"ts":"1676127621193"} 2023-02-11 15:00:21,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:21,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:21,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:21,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:21,194 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:21,195 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:21,195 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:21,196 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:21,196 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:21,196 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:21,196 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:21,197 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cd2163f692eb299f43cb55be4d4dc49f columnFamilyName cf 2023-02-11 15:00:21,197 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:21,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=123 2023-02-11 15:00:21,197 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:21,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=123, state=SUCCESS; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743 in 163 msec 2023-02-11 15:00:21,197 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78f87f7851c87dde38a4d94bc805db9b columnFamilyName cf 2023-02-11 15:00:21,197 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7677bc32f6d3f7225f4cfdeb2068fc26 columnFamilyName cf 2023-02-11 15:00:21,198 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=125 2023-02-11 15:00:21,198 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(310): Store=cd2163f692eb299f43cb55be4d4dc49f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:21,198 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=125, state=SUCCESS; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785 in 169 msec 2023-02-11 15:00:21,198 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=124 2023-02-11 15:00:21,198 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(310): Store=78f87f7851c87dde38a4d94bc805db9b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:21,198 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=124, state=SUCCESS; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,41207,1676127604675 in 167 msec 2023-02-11 15:00:21,199 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN in 332 msec 2023-02-11 15:00:21,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:21,199 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(310): Store=7677bc32f6d3f7225f4cfdeb2068fc26/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:21,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:21,200 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN in 333 msec 2023-02-11 15:00:21,200 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN in 333 msec 2023-02-11 15:00:21,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:21,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:21,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:21,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:21,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:21,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:21,203 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 78f87f7851c87dde38a4d94bc805db9b; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71368620, jitterRate=0.06347531080245972}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:21,203 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened cd2163f692eb299f43cb55be4d4dc49f; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64585867, jitterRate=-0.037595584988594055}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:21,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 78f87f7851c87dde38a4d94bc805db9b: Running coprocessor pre-open hook at 1676127621193Writing region info on filesystem at 1676127621193Initializing all the Stores at 1676127621194 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127621194Cleaning up temporary data from old regions at 1676127621200 (+6 ms)Cleaning up detritus from prior splits at 1676127621201 (+1 ms)Running coprocessor post-open hooks at 1676127621203 (+2 ms)Region opened successfully at 1676127621203 2023-02-11 15:00:21,203 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7677bc32f6d3f7225f4cfdeb2068fc26; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71221626, jitterRate=0.061284929513931274}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:21,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for cd2163f692eb299f43cb55be4d4dc49f: Running coprocessor pre-open hook at 1676127621193Writing region info on filesystem at 1676127621193Initializing all the Stores at 1676127621194 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127621195 (+1 ms)Cleaning up temporary data from old regions at 1676127621201 (+6 ms)Cleaning up detritus from prior splits at 1676127621201Running coprocessor post-open hooks at 1676127621203 (+2 ms)Region opened successfully at 1676127621203 2023-02-11 15:00:21,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Running coprocessor pre-open hook at 1676127621194Writing region info on filesystem at 1676127621194Initializing all the Stores at 1676127621195 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127621195Cleaning up temporary data from old regions at 1676127621201 (+6 ms)Cleaning up detritus from prior splits at 1676127621201Running coprocessor post-open hooks at 1676127621203 (+2 ms)Region opened successfully at 1676127621203 2023-02-11 15:00:21,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b., pid=129, masterSystemTime=1676127621178 2023-02-11 15:00:21,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f., pid=128, masterSystemTime=1676127621177 2023-02-11 15:00:21,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26., pid=127, masterSystemTime=1676127621177 2023-02-11 15:00:21,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:21,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:21,205 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:21,206 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127621205"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127621205"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127621205"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127621205"}]},"ts":"1676127621205"} 2023-02-11 15:00:21,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:21,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:21,206 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:21,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:21,206 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127621206"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127621206"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127621206"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127621206"}]},"ts":"1676127621206"} 2023-02-11 15:00:21,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:21,207 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:21,207 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621207"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127621207"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127621207"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127621207"}]},"ts":"1676127621207"} 2023-02-11 15:00:21,209 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=126 2023-02-11 15:00:21,209 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=126, state=SUCCESS; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743 in 180 msec 2023-02-11 15:00:21,210 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=121 2023-02-11 15:00:21,210 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=121, state=SUCCESS; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,41207,1676127604675 in 183 msec 2023-02-11 15:00:21,210 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN in 344 msec 2023-02-11 15:00:21,211 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=122 2023-02-11 15:00:21,211 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=122, state=SUCCESS; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,42505,1676127604785 in 183 msec 2023-02-11 15:00:21,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN in 345 msec 2023-02-11 15:00:21,212 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=122, resume processing ppid=120 2023-02-11 15:00:21,212 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN in 346 msec 2023-02-11 15:00:21,213 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127621213"}]},"ts":"1676127621213"} 2023-02-11 15:00:21,214 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLED in hbase:meta 2023-02-11 15:00:21,216 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676127607597' was successfully enabled. 2023-02-11 15:00:21,217 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676127607597 in 358 msec 2023-02-11 15:00:21,517 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 15:00:21,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 15:00:21,613 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 120 completed 2023-02-11 15:00:21,613 DEBUG [Listener at localhost/39605] zookeeper.ReadOnlyZKClient(139): Connect 0x5f3a7cee to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:21,619 DEBUG [Listener at localhost/39605] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43ec0260, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:21,620 DEBUG [hconnection-0x2606f9a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:21,623 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42476, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:21,627 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 15:00:21,629 DEBUG [hconnection-0x2606f9a8-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:21,631 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34118, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:21,632 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 15:00:21,634 DEBUG [hconnection-0x2606f9a8-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:21,636 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38700, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:21,637 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 15:00:21,641 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 15:00:21,644 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 15:00:21,650 INFO [Listener at localhost/39605] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:21,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676127607597 2023-02-11 15:00:21,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:21,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-11 15:00:21,654 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127621654"}]},"ts":"1676127621654"} 2023-02-11 15:00:21,655 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLING in hbase:meta 2023-02-11 15:00:21,656 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676127607597 to state=DISABLING 2023-02-11 15:00:21,657 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN}] 2023-02-11 15:00:21,659 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN 2023-02-11 15:00:21,659 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN 2023-02-11 15:00:21,659 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN 2023-02-11 15:00:21,659 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN 2023-02-11 15:00:21,659 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN 2023-02-11 15:00:21,660 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:21,660 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621660"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621660"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621660"}]},"ts":"1676127621660"} 2023-02-11 15:00:21,660 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:21,660 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:21,660 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621660"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621660"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621660"}]},"ts":"1676127621660"} 2023-02-11 15:00:21,660 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621660"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621660"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621660"}]},"ts":"1676127621660"} 2023-02-11 15:00:21,660 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:21,660 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:21,661 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127621660"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621660"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621660"}]},"ts":"1676127621660"} 2023-02-11 15:00:21,661 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127621660"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621660"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621660"}]},"ts":"1676127621660"} 2023-02-11 15:00:21,661 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=136, state=RUNNABLE; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:21,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=137, state=RUNNABLE; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:21,663 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=135, state=RUNNABLE; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:21,664 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=139, state=RUNNABLE; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:21,665 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN 2023-02-11 15:00:21,665 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=134, state=RUNNABLE; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:21,665 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:21,666 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621665"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127621665"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127621665"}]},"ts":"1676127621665"} 2023-02-11 15:00:21,667 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=138, state=RUNNABLE; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:21,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:21,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 78f87f7851c87dde38a4d94bc805db9b, disabling compactions & flushes 2023-02-11 15:00:21,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:21,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:21,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. after waiting 0 ms 2023-02-11 15:00:21,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:21,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:21,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:21,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7677bc32f6d3f7225f4cfdeb2068fc26, disabling compactions & flushes 2023-02-11 15:00:21,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing cd2163f692eb299f43cb55be4d4dc49f, disabling compactions & flushes 2023-02-11 15:00:21,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:21,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:21,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:21,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. after waiting 0 ms 2023-02-11 15:00:21,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:21,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:21,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. after waiting 0 ms 2023-02-11 15:00:21,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:21,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 15:00:21,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 15:00:21,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 15:00:21,821 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:21,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 78f87f7851c87dde38a4d94bc805db9b: Waiting for close lock at 1676127621814Running coprocessor pre-close hooks at 1676127621814Disabling compacts and flushes for region at 1676127621814Disabling writes for close at 1676127621815 (+1 ms)Writing region close event to WAL at 1676127621815Running coprocessor post-close hooks at 1676127621821 (+6 ms)Closed at 1676127621821 2023-02-11 15:00:21,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:21,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Waiting for close lock at 1676127621815Running coprocessor pre-close hooks at 1676127621815Disabling compacts and flushes for region at 1676127621815Disabling writes for close at 1676127621817 (+2 ms)Writing region close event to WAL at 1676127621818 (+1 ms)Running coprocessor post-close hooks at 1676127621821 (+3 ms)Closed at 1676127621822 (+1 ms) 2023-02-11 15:00:21,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:21,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for cd2163f692eb299f43cb55be4d4dc49f: Waiting for close lock at 1676127621816Running coprocessor pre-close hooks at 1676127621816Disabling compacts and flushes for region at 1676127621816Disabling writes for close at 1676127621817 (+1 ms)Writing region close event to WAL at 1676127621818 (+1 ms)Running coprocessor post-close hooks at 1676127621822 (+4 ms)Closed at 1676127621823 (+1 ms) 2023-02-11 15:00:21,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:21,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:21,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 704f07b3665f9baa8c5a5484e3bdbcd6, disabling compactions & flushes 2023-02-11 15:00:21,825 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:21,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:21,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. after waiting 0 ms 2023-02-11 15:00:21,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:21,825 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSED 2023-02-11 15:00:21,825 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127621825"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127621825"}]},"ts":"1676127621825"} 2023-02-11 15:00:21,825 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:21,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:21,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2c301b6be53a2e3d2c7174a30417ca8e, disabling compactions & flushes 2023-02-11 15:00:21,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:21,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:21,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. after waiting 0 ms 2023-02-11 15:00:21,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:21,827 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSED 2023-02-11 15:00:21,827 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127621827"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127621827"}]},"ts":"1676127621827"} 2023-02-11 15:00:21,827 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:21,827 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:21,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 94a0d6907c18dd6eadf539b9aa0231e2, disabling compactions & flushes 2023-02-11 15:00:21,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:21,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:21,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. after waiting 0 ms 2023-02-11 15:00:21,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:21,828 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSED 2023-02-11 15:00:21,828 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621828"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127621828"}]},"ts":"1676127621828"} 2023-02-11 15:00:21,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 15:00:21,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:21,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Waiting for close lock at 1676127621824Running coprocessor pre-close hooks at 1676127621824Disabling compacts and flushes for region at 1676127621824Disabling writes for close at 1676127621825 (+1 ms)Writing region close event to WAL at 1676127621825Running coprocessor post-close hooks at 1676127621830 (+5 ms)Closed at 1676127621831 (+1 ms) 2023-02-11 15:00:21,832 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=139 2023-02-11 15:00:21,832 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=139, state=SUCCESS; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743 in 163 msec 2023-02-11 15:00:21,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:21,833 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=134 2023-02-11 15:00:21,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 15:00:21,833 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=134, state=SUCCESS; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,41207,1676127604675 in 165 msec 2023-02-11 15:00:21,833 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSED 2023-02-11 15:00:21,833 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621833"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127621833"}]},"ts":"1676127621833"} 2023-02-11 15:00:21,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:21,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 94a0d6907c18dd6eadf539b9aa0231e2: Waiting for close lock at 1676127621827Running coprocessor pre-close hooks at 1676127621827Disabling compacts and flushes for region at 1676127621827Disabling writes for close at 1676127621828 (+1 ms)Writing region close event to WAL at 1676127621828Running coprocessor post-close hooks at 1676127621833 (+5 ms)Closed at 1676127621834 (+1 ms) 2023-02-11 15:00:21,834 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN in 175 msec 2023-02-11 15:00:21,835 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=135 2023-02-11 15:00:21,835 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=135, state=SUCCESS; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,42505,1676127604785 in 169 msec 2023-02-11 15:00:21,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 15:00:21,835 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN in 176 msec 2023-02-11 15:00:21,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:21,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:21,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2c301b6be53a2e3d2c7174a30417ca8e: Waiting for close lock at 1676127621826Running coprocessor pre-close hooks at 1676127621826Disabling compacts and flushes for region at 1676127621826Disabling writes for close at 1676127621826Writing region close event to WAL at 1676127621828 (+2 ms)Running coprocessor post-close hooks at 1676127621835 (+7 ms)Closed at 1676127621836 (+1 ms) 2023-02-11 15:00:21,836 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSED 2023-02-11 15:00:21,836 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621836"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127621836"}]},"ts":"1676127621836"} 2023-02-11 15:00:21,837 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN in 178 msec 2023-02-11 15:00:21,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:21,838 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSED 2023-02-11 15:00:21,838 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127621838"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127621838"}]},"ts":"1676127621838"} 2023-02-11 15:00:21,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=136 2023-02-11 15:00:21,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=136, state=SUCCESS; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743 in 174 msec 2023-02-11 15:00:21,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=138 2023-02-11 15:00:21,839 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN in 181 msec 2023-02-11 15:00:21,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=138, state=SUCCESS; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,42505,1676127604785 in 171 msec 2023-02-11 15:00:21,840 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN in 182 msec 2023-02-11 15:00:21,843 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=137 2023-02-11 15:00:21,843 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=137, state=SUCCESS; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,41207,1676127604675 in 180 msec 2023-02-11 15:00:21,844 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=137, resume processing ppid=133 2023-02-11 15:00:21,844 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN in 186 msec 2023-02-11 15:00:21,845 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127621845"}]},"ts":"1676127621845"} 2023-02-11 15:00:21,846 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLED in hbase:meta 2023-02-11 15:00:21,849 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676127607597 to state=DISABLED 2023-02-11 15:00:21,851 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676127607597 in 199 msec 2023-02-11 15:00:21,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-11 15:00:21,905 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 133 completed 2023-02-11 15:00:21,914 INFO [Listener at localhost/39605] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676127607597-1676127621914 2023-02-11 15:00:21,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676127607597-1676127621914 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:21,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676127621914 (current time:1676127621914). 2023-02-11 15:00:21,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 15:00:21,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(139): Connect 0x7c731f1d to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:21,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a38e3a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:21,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:21,923 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42490, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:21,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c731f1d to 127.0.0.1:58940 2023-02-11 15:00:21,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:21,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 15:00:21,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676127607597-1676127621914 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:21,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:21,927 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:21,927 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE; timeout=600000, timestamp=1676128221927 2023-02-11 15:00:21,927 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676127607597-1676127621914 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 2023-02-11 15:00:21,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676127607597-1676127621914 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } 2023-02-11 15:00:21,928 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676127607597-1676127621914 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 15:00:21,928 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 15:00:21,937 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676127607597-1676127621914 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } 2023-02-11 15:00:21,938 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:21,938 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:21,938 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:21,938 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:21,938 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:21,938 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:21,939 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:21,939 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:21,939 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:21,939 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:21,939 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:21,939 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:21,939 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:21,939 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:21,939 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:21,940 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:21,940 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:21,941 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:21,941 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:21,941 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:21,941 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:21,963 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676127607597-1676127621914 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } as finished. 2023-02-11 15:00:21,963 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676127607597-1676127621914 2023-02-11 15:00:21,963 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676127607597-1676127621914 2023-02-11 15:00:21,980 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676127607597-1676127621914 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/hbase-failsafe-snaptb1-1676127607597-1676127621914 2023-02-11 15:00:21,995 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676127607597-1676127621914 of table testRestoreSnapshot-1676127607597 completed 2023-02-11 15:00:21,995 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676127607597-1676127621914 2023-02-11 15:00:21,996 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676127607597-1676127621914 2023-02-11 15:00:21,996 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676127607597-1676127621914 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676127607597 at 1676127621927Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676127607597-1676127621914 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } at 1676127621937 (+10 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676127607597-1676127621914 at 1676127621963 (+26 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676127607597-1676127621914 at 1676127621963Writing data manifest for hbase-failsafe-snaptb1-1676127607597-1676127621914 at 1676127621967 (+4 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676127607597-1676127621914 at 1676127621974 (+7 ms)Snapshot hbase-failsafe-snaptb1-1676127607597-1676127621914 of table testRestoreSnapshot-1676127607597 completed at 1676127621995 (+21 ms) 2023-02-11 15:00:21,998 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE 2023-02-11 15:00:21,999 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676127607597, type=EXCLUSIVE in 73 msec 2023-02-11 15:00:22,178 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 15:00:22,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676127607597-1676127621914 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 } is done 2023-02-11 15:00:22,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676127607597-1676127621914 table=testRestoreSnapshot-1676127607597 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 15:00:22,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676127607597 snapshot=name: "snaptb1-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127616534 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 15:00:22,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676127607597 as table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-11 15:00:22,206 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000004 2023-02-11 15:00:22,207 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000003 2023-02-11 15:00:22,207 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000004 2023-02-11 15:00:22,208 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } 2023-02-11 15:00:22,210 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127616534 type: DISABLED version: 2 ttl: 0 2023-02-11 15:00:22,210 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,217 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,218 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,222 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,225 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,229 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,233 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,236 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,240 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,243 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,247 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,250 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,253 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,257 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 of snapshot=snaptb1-1676127607597 to region=be8dfc6e3907aa03a0317253d205c628 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,260 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:22,260 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:22,260 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:22,260 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:22,260 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:22,261 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:22,262 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 2bab4cc59a7d4614827ba4ccb10f3bd6 of snapshot=snaptb1-1676127607597 to region=2c301b6be53a2e3d2c7174a30417ca8e table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,262 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 830bf232363e40bbbca42f0cf5644c1d of snapshot=snaptb1-1676127607597 to region=704f07b3665f9baa8c5a5484e3bdbcd6 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,262 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 08c52f6c5115434fa8273a8c2162f5e1 of snapshot=snaptb1-1676127607597 to region=94a0d6907c18dd6eadf539b9aa0231e2 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,262 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 46fbcfe2640b44ad951ffc998805ea49 of snapshot=snaptb1-1676127607597 to region=78f87f7851c87dde38a4d94bc805db9b table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,263 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 80b2a4abbc984f83a8efd755415eaf73 of snapshot=snaptb1-1676127607597 to region=7677bc32f6d3f7225f4cfdeb2068fc26 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,263 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 2a35e2af579a4f78aad755c9c1a42e33 of snapshot=snaptb1-1676127607597 to region=cd2163f692eb299f43cb55be4d4dc49f table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,272 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 49a06ed7ffdc4dc9a2446e463bc2be02 of snapshot=snaptb1-1676127607597 to region=94a0d6907c18dd6eadf539b9aa0231e2 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,272 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a11a6bb05820487da1f0bf1a178b54e0 of snapshot=snaptb1-1676127607597 to region=704f07b3665f9baa8c5a5484e3bdbcd6 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,272 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink ef1ac041d7184ad1af9b3666b7d994ae of snapshot=snaptb1-1676127607597 to region=7677bc32f6d3f7225f4cfdeb2068fc26 table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,273 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 4fefa1144af6408195c155495f67763b of snapshot=snaptb1-1676127607597 to region=cd2163f692eb299f43cb55be4d4dc49f table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,273 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink cf109e42ce0a4e698dd14c8965b4c03a of snapshot=snaptb1-1676127607597 to region=78f87f7851c87dde38a4d94bc805db9b table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,273 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink ea92c7cd9a554d4a849955a308ed7e13 of snapshot=snaptb1-1676127607597 to region=2c301b6be53a2e3d2c7174a30417ca8e table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,289 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127616534 type: DISABLED version: 2 ttl: 0 2023-02-11 15:00:22,290 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127622290"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:22,290 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127622290"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:22,290 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127622290"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:22,290 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127622290"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:22,290 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127622290"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:22,290 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127622290"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:22,292 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 15:00:22,292 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 15:00:22,293 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622291"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127622291"}]},"ts":"1676127622291"} 2023-02-11 15:00:22,293 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622291"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127622291"}]},"ts":"1676127622291"} 2023-02-11 15:00:22,293 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127622291"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127622291"}]},"ts":"1676127622291"} 2023-02-11 15:00:22,293 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127622291"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127622291"}]},"ts":"1676127622291"} 2023-02-11 15:00:22,293 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622291"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127622291"}]},"ts":"1676127622291"} 2023-02-11 15:00:22,293 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622291"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127622291"}]},"ts":"1676127622291"} 2023-02-11 15:00:22,294 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 15:00:22,294 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 15:00:22,294 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-11 15:00:22,294 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676127607597 table=testRestoreSnapshot-1676127607597 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676127607597 completed! 2023-02-11 15:00:22,296 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676127607597 snapshot=name: "snaptb1-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127616534 type: DISABLED version: 2 ttl: 0 ) in 104 msec 2023-02-11 15:00:22,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-11 15:00:22,447 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676127607597, procId: 147 completed 2023-02-11 15:00:22,447 INFO [Listener at localhost/39605] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676127607597-1676127621914 2023-02-11 15:00:22,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1676127607597-1676127621914" 2023-02-11 15:00:22,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676127607597-1676127621914 2023-02-11 15:00:22,451 INFO [Listener at localhost/39605] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 15:00:22,454 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676127607597 2023-02-11 15:00:22,454 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127622454"}]},"ts":"1676127622454"} 2023-02-11 15:00:22,456 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLING in hbase:meta 2023-02-11 15:00:22,459 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 15:00:22,459 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN}] 2023-02-11 15:00:22,461 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN 2023-02-11 15:00:22,461 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN 2023-02-11 15:00:22,461 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN 2023-02-11 15:00:22,461 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN 2023-02-11 15:00:22,462 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:22,462 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:22,462 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:22,462 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:22,462 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN 2023-02-11 15:00:22,463 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN 2023-02-11 15:00:22,463 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:22,463 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 15:00:22,612 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 15:00:22,612 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 15:00:22,612 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 15:00:22,612 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 15:00:22,612 DEBUG [jenkins-hbase4:38517] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 15:00:22,615 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:22,615 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:22,615 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:22,615 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:22,616 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127622615"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127622615"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127622615"}]},"ts":"1676127622615"} 2023-02-11 15:00:22,616 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622615"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127622615"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127622615"}]},"ts":"1676127622615"} 2023-02-11 15:00:22,615 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:22,616 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127622615"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127622615"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127622615"}]},"ts":"1676127622615"} 2023-02-11 15:00:22,616 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622615"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127622615"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127622615"}]},"ts":"1676127622615"} 2023-02-11 15:00:22,616 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622615"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127622615"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127622615"}]},"ts":"1676127622615"} 2023-02-11 15:00:22,617 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=154, state=RUNNABLE; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:22,618 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=153, state=RUNNABLE; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:22,619 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=149, state=RUNNABLE; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:22,620 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=151, state=RUNNABLE; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:22,621 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=152, state=RUNNABLE; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:22,621 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:22,621 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622621"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127622621"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127622621"}]},"ts":"1676127622621"} 2023-02-11 15:00:22,622 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=150, state=RUNNABLE; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:22,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 15:00:22,772 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:22,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 15:00:22,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:22,772 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:22,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:22,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 15:00:22,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:22,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:22,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:22,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:22,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:22,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:22,774 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:22,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:22,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 15:00:22,774 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:22,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:22,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:22,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:22,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:22,774 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:22,775 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:22,775 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:22,775 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 704f07b3665f9baa8c5a5484e3bdbcd6 columnFamilyName cf 2023-02-11 15:00:22,775 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:22,775 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:22,775 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94a0d6907c18dd6eadf539b9aa0231e2 columnFamilyName cf 2023-02-11 15:00:22,776 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:22,776 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:22,776 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7677bc32f6d3f7225f4cfdeb2068fc26 columnFamilyName cf 2023-02-11 15:00:22,791 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:22,791 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-08c52f6c5115434fa8273a8c2162f5e1 2023-02-11 15:00:22,791 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-80b2a4abbc984f83a8efd755415eaf73 2023-02-11 15:00:22,798 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-a11a6bb05820487da1f0bf1a178b54e0 2023-02-11 15:00:22,798 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(310): Store=704f07b3665f9baa8c5a5484e3bdbcd6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:22,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:22,800 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:22,800 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(310): Store=7677bc32f6d3f7225f4cfdeb2068fc26/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:22,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:22,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:22,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:22,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:22,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:22,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 704f07b3665f9baa8c5a5484e3bdbcd6; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70037156, jitterRate=0.04363495111465454}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:22,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Running coprocessor pre-open hook at 1676127622773Writing region info on filesystem at 1676127622773Initializing all the Stores at 1676127622773Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127622773Cleaning up temporary data from old regions at 1676127622801 (+28 ms)Cleaning up detritus from prior splits at 1676127622802 (+1 ms)Running coprocessor post-open hooks at 1676127622805 (+3 ms)Region opened successfully at 1676127622805 2023-02-11 15:00:22,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6., pid=158, masterSystemTime=1676127622769 2023-02-11 15:00:22,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7677bc32f6d3f7225f4cfdeb2068fc26; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58916472, jitterRate=-0.12207615375518799}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:22,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Running coprocessor pre-open hook at 1676127622774Writing region info on filesystem at 1676127622774Initializing all the Stores at 1676127622775 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127622775Cleaning up temporary data from old regions at 1676127622803 (+28 ms)Cleaning up detritus from prior splits at 1676127622803Running coprocessor post-open hooks at 1676127622806 (+3 ms)Region opened successfully at 1676127622806 2023-02-11 15:00:22,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26., pid=157, masterSystemTime=1676127622771 2023-02-11 15:00:22,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:22,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:22,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:22,808 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:22,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 15:00:22,808 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(310): Store=94a0d6907c18dd6eadf539b9aa0231e2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:22,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:22,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:22,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:22,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:22,808 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:22,809 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622808"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127622808"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127622808"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127622808"}]},"ts":"1676127622808"} 2023-02-11 15:00:22,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:22,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:22,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:22,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 15:00:22,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:22,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:22,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:22,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:22,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:22,810 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:22,810 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127622810"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127622810"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127622810"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127622810"}]},"ts":"1676127622810"} 2023-02-11 15:00:22,811 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:22,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:22,812 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:22,812 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:22,812 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:22,813 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=151 2023-02-11 15:00:22,813 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=151, state=SUCCESS; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675 in 190 msec 2023-02-11 15:00:22,813 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c301b6be53a2e3d2c7174a30417ca8e columnFamilyName cf 2023-02-11 15:00:22,813 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:22,813 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:22,814 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78f87f7851c87dde38a4d94bc805db9b columnFamilyName cf 2023-02-11 15:00:22,815 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=149 2023-02-11 15:00:22,815 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN in 354 msec 2023-02-11 15:00:22,815 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=149, state=SUCCESS; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785 in 193 msec 2023-02-11 15:00:22,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:22,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 94a0d6907c18dd6eadf539b9aa0231e2; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66248906, jitterRate=-0.012814372777938843}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:22,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 94a0d6907c18dd6eadf539b9aa0231e2: Running coprocessor pre-open hook at 1676127622773Writing region info on filesystem at 1676127622773Initializing all the Stores at 1676127622774 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127622774Cleaning up temporary data from old regions at 1676127622812 (+38 ms)Cleaning up detritus from prior splits at 1676127622813 (+1 ms)Running coprocessor post-open hooks at 1676127622816 (+3 ms)Region opened successfully at 1676127622816 2023-02-11 15:00:22,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN in 356 msec 2023-02-11 15:00:22,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2., pid=156, masterSystemTime=1676127622770 2023-02-11 15:00:22,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:22,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:22,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:22,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 15:00:22,820 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:22,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:22,820 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622819"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127622819"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127622819"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127622819"}]},"ts":"1676127622819"} 2023-02-11 15:00:22,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:22,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:22,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:22,822 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:22,823 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:22,823 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:22,825 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cd2163f692eb299f43cb55be4d4dc49f columnFamilyName cf 2023-02-11 15:00:22,825 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=153 2023-02-11 15:00:22,825 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=153, state=SUCCESS; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,33165,1676127604743 in 204 msec 2023-02-11 15:00:22,825 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:22,825 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-46fbcfe2640b44ad951ffc998805ea49 2023-02-11 15:00:22,826 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN in 366 msec 2023-02-11 15:00:22,838 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:22,838 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-ea92c7cd9a554d4a849955a308ed7e13 2023-02-11 15:00:22,838 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(310): Store=78f87f7851c87dde38a4d94bc805db9b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:22,838 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(310): Store=2c301b6be53a2e3d2c7174a30417ca8e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:22,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:22,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:22,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:22,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:22,841 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:22,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:22,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:22,844 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 78f87f7851c87dde38a4d94bc805db9b; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68951724, jitterRate=0.027460753917694092}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:22,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 78f87f7851c87dde38a4d94bc805db9b: Running coprocessor pre-open hook at 1676127622809Writing region info on filesystem at 1676127622809Initializing all the Stores at 1676127622811 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127622811Cleaning up temporary data from old regions at 1676127622841 (+30 ms)Cleaning up detritus from prior splits at 1676127622842 (+1 ms)Running coprocessor post-open hooks at 1676127622844 (+2 ms)Region opened successfully at 1676127622844 2023-02-11 15:00:22,844 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2c301b6be53a2e3d2c7174a30417ca8e; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60277430, jitterRate=-0.10179629921913147}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:22,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2c301b6be53a2e3d2c7174a30417ca8e: Running coprocessor pre-open hook at 1676127622810Writing region info on filesystem at 1676127622810Initializing all the Stores at 1676127622811 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127622811Cleaning up temporary data from old regions at 1676127622841 (+30 ms)Cleaning up detritus from prior splits at 1676127622842 (+1 ms)Running coprocessor post-open hooks at 1676127622844 (+2 ms)Region opened successfully at 1676127622844 2023-02-11 15:00:22,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b., pid=155, masterSystemTime=1676127622769 2023-02-11 15:00:22,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e., pid=159, masterSystemTime=1676127622771 2023-02-11 15:00:22,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:22,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:22,848 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:22,848 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127622847"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127622847"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127622847"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127622847"}]},"ts":"1676127622847"} 2023-02-11 15:00:22,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:22,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:22,848 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:22,848 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622848"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127622848"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127622848"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127622848"}]},"ts":"1676127622848"} 2023-02-11 15:00:22,851 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-4fefa1144af6408195c155495f67763b 2023-02-11 15:00:22,851 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(310): Store=cd2163f692eb299f43cb55be4d4dc49f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:22,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:22,853 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=154 2023-02-11 15:00:22,853 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=154, state=SUCCESS; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675 in 234 msec 2023-02-11 15:00:22,853 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=152 2023-02-11 15:00:22,854 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=152, state=SUCCESS; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,42505,1676127604785 in 230 msec 2023-02-11 15:00:22,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:22,854 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN in 394 msec 2023-02-11 15:00:22,855 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN in 395 msec 2023-02-11 15:00:22,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:22,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened cd2163f692eb299f43cb55be4d4dc49f; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64756233, jitterRate=-0.035056933760643005}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:22,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for cd2163f692eb299f43cb55be4d4dc49f: Running coprocessor pre-open hook at 1676127622821Writing region info on filesystem at 1676127622821Initializing all the Stores at 1676127622821Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127622821Cleaning up temporary data from old regions at 1676127622854 (+33 ms)Cleaning up detritus from prior splits at 1676127622855 (+1 ms)Running coprocessor post-open hooks at 1676127622857 (+2 ms)Region opened successfully at 1676127622857 2023-02-11 15:00:22,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f., pid=160, masterSystemTime=1676127622770 2023-02-11 15:00:22,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:22,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:22,859 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:22,859 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127622859"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127622859"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127622859"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127622859"}]},"ts":"1676127622859"} 2023-02-11 15:00:22,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=150 2023-02-11 15:00:22,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=150, state=SUCCESS; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743 in 239 msec 2023-02-11 15:00:22,864 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=150, resume processing ppid=148 2023-02-11 15:00:22,864 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN in 403 msec 2023-02-11 15:00:22,865 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127622864"}]},"ts":"1676127622864"} 2023-02-11 15:00:22,866 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLED in hbase:meta 2023-02-11 15:00:22,874 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676127607597' was successfully enabled. 2023-02-11 15:00:22,875 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676127607597 in 423 msec 2023-02-11 15:00:23,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 15:00:23,206 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 148 completed 2023-02-11 15:00:23,206 DEBUG [Listener at localhost/39605] zookeeper.ReadOnlyZKClient(139): Connect 0x1550b85e to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:23,212 DEBUG [Listener at localhost/39605] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ed0e7fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:23,213 DEBUG [hconnection-0x70b95fe7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:23,216 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42494, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:23,219 DEBUG [hconnection-0x70b95fe7-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:23,221 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34128, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:23,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,409 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 15:00:23,412 DEBUG [hconnection-0x70b95fe7-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:23,414 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54240, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:23,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,489 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x272618c8 to 127.0.0.1:58940 2023-02-11 15:00:23,489 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:23,491 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f3a7cee to 127.0.0.1:58940 2023-02-11 15:00:23,491 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:23,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,614 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 15:00:23,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,849 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 15:00:23,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:23,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,070 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 15:00:24,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,098 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676127607597' 2023-02-11 15:00:24,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,253 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 15:00:24,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:24,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:25,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:26,319 INFO [Listener at localhost/39605] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-11 15:00:26,324 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127626324"}]},"ts":"1676127626324"} 2023-02-11 15:00:26,325 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLING in hbase:meta 2023-02-11 15:00:26,332 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676127607597 to state=DISABLING 2023-02-11 15:00:26,332 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN}] 2023-02-11 15:00:26,334 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN 2023-02-11 15:00:26,334 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN 2023-02-11 15:00:26,334 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN 2023-02-11 15:00:26,335 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN 2023-02-11 15:00:26,335 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN 2023-02-11 15:00:26,335 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:26,335 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:26,335 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127626335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127626335"}]},"ts":"1676127626335"} 2023-02-11 15:00:26,335 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127626335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127626335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127626335"}]},"ts":"1676127626335"} 2023-02-11 15:00:26,335 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:26,335 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626335"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127626335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127626335"}]},"ts":"1676127626335"} 2023-02-11 15:00:26,336 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:26,336 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:26,336 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127626336"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127626336"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127626336"}]},"ts":"1676127626336"} 2023-02-11 15:00:26,336 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626336"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127626336"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127626336"}]},"ts":"1676127626336"} 2023-02-11 15:00:26,337 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=163, state=RUNNABLE; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:26,337 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=162, state=RUNNABLE; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:26,338 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=166, state=RUNNABLE; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:26,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=167, state=RUNNABLE; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:26,339 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=165, state=RUNNABLE; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:26,339 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN 2023-02-11 15:00:26,340 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:26,340 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626340"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127626340"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127626340"}]},"ts":"1676127626340"} 2023-02-11 15:00:26,341 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=164, state=RUNNABLE; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:26,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:26,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing cd2163f692eb299f43cb55be4d4dc49f, disabling compactions & flushes 2023-02-11 15:00:26,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:26,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:26,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. after waiting 0 ms 2023-02-11 15:00:26,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:26,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:26,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:26,491 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2c301b6be53a2e3d2c7174a30417ca8e, disabling compactions & flushes 2023-02-11 15:00:26,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:26,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:26,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. after waiting 0 ms 2023-02-11 15:00:26,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:26,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 78f87f7851c87dde38a4d94bc805db9b, disabling compactions & flushes 2023-02-11 15:00:26,494 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:26,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:26,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. after waiting 0 ms 2023-02-11 15:00:26,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:26,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 15:00:26,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 15:00:26,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 15:00:26,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:26,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:26,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:26,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 78f87f7851c87dde38a4d94bc805db9b: Waiting for close lock at 1676127626492Running coprocessor pre-close hooks at 1676127626492Disabling compacts and flushes for region at 1676127626492Disabling writes for close at 1676127626495 (+3 ms)Writing region close event to WAL at 1676127626501 (+6 ms)Running coprocessor post-close hooks at 1676127626506 (+5 ms)Closed at 1676127626506 2023-02-11 15:00:26,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2c301b6be53a2e3d2c7174a30417ca8e: Waiting for close lock at 1676127626490Running coprocessor pre-close hooks at 1676127626490Disabling compacts and flushes for region at 1676127626490Disabling writes for close at 1676127626492 (+2 ms)Writing region close event to WAL at 1676127626498 (+6 ms)Running coprocessor post-close hooks at 1676127626506 (+8 ms)Closed at 1676127626507 (+1 ms) 2023-02-11 15:00:26,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for cd2163f692eb299f43cb55be4d4dc49f: Waiting for close lock at 1676127626489Running coprocessor pre-close hooks at 1676127626489Disabling compacts and flushes for region at 1676127626489Disabling writes for close at 1676127626490 (+1 ms)Writing region close event to WAL at 1676127626496 (+6 ms)Running coprocessor post-close hooks at 1676127626506 (+10 ms)Closed at 1676127626506 2023-02-11 15:00:26,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:26,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:26,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 94a0d6907c18dd6eadf539b9aa0231e2, disabling compactions & flushes 2023-02-11 15:00:26,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:26,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:26,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. after waiting 0 ms 2023-02-11 15:00:26,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:26,509 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSED 2023-02-11 15:00:26,509 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626509"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626509"}]},"ts":"1676127626509"} 2023-02-11 15:00:26,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:26,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:26,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7677bc32f6d3f7225f4cfdeb2068fc26, disabling compactions & flushes 2023-02-11 15:00:26,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:26,511 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSED 2023-02-11 15:00:26,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:26,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. after waiting 0 ms 2023-02-11 15:00:26,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:26,513 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626511"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626511"}]},"ts":"1676127626511"} 2023-02-11 15:00:26,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:26,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:26,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 704f07b3665f9baa8c5a5484e3bdbcd6, disabling compactions & flushes 2023-02-11 15:00:26,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:26,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:26,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. after waiting 0 ms 2023-02-11 15:00:26,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:26,514 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSED 2023-02-11 15:00:26,514 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127626514"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626514"}]},"ts":"1676127626514"} 2023-02-11 15:00:26,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 15:00:26,529 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=163 2023-02-11 15:00:26,529 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=163, state=SUCCESS; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,33165,1676127604743 in 181 msec 2023-02-11 15:00:26,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 15:00:26,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:26,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Waiting for close lock at 1676127626510Running coprocessor pre-close hooks at 1676127626510Disabling compacts and flushes for region at 1676127626510Disabling writes for close at 1676127626513 (+3 ms)Writing region close event to WAL at 1676127626518 (+5 ms)Running coprocessor post-close hooks at 1676127626529 (+11 ms)Closed at 1676127626529 2023-02-11 15:00:26,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:26,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 94a0d6907c18dd6eadf539b9aa0231e2: Waiting for close lock at 1676127626508Running coprocessor pre-close hooks at 1676127626508Disabling compacts and flushes for region at 1676127626508Disabling writes for close at 1676127626509 (+1 ms)Writing region close event to WAL at 1676127626513 (+4 ms)Running coprocessor post-close hooks at 1676127626529 (+16 ms)Closed at 1676127626530 (+1 ms) 2023-02-11 15:00:26,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 15:00:26,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:26,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Waiting for close lock at 1676127626513Running coprocessor pre-close hooks at 1676127626513Disabling compacts and flushes for region at 1676127626513Disabling writes for close at 1676127626514 (+1 ms)Writing region close event to WAL at 1676127626528 (+14 ms)Running coprocessor post-close hooks at 1676127626532 (+4 ms)Closed at 1676127626532 2023-02-11 15:00:26,532 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=165 2023-02-11 15:00:26,532 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=165, state=SUCCESS; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,42505,1676127604785 in 182 msec 2023-02-11 15:00:26,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:26,533 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=167 2023-02-11 15:00:26,533 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN in 197 msec 2023-02-11 15:00:26,533 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSED 2023-02-11 15:00:26,533 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=167, state=SUCCESS; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,41207,1676127604675 in 186 msec 2023-02-11 15:00:26,534 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127626533"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626533"}]},"ts":"1676127626533"} 2023-02-11 15:00:26,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:26,534 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN in 200 msec 2023-02-11 15:00:26,534 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSED 2023-02-11 15:00:26,534 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626534"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626534"}]},"ts":"1676127626534"} 2023-02-11 15:00:26,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:26,535 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN in 201 msec 2023-02-11 15:00:26,535 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSED 2023-02-11 15:00:26,535 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626535"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626535"}]},"ts":"1676127626535"} 2023-02-11 15:00:26,537 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=162 2023-02-11 15:00:26,537 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=162, state=SUCCESS; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785 in 198 msec 2023-02-11 15:00:26,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=166 2023-02-11 15:00:26,538 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=166, state=SUCCESS; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,33165,1676127604743 in 198 msec 2023-02-11 15:00:26,538 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=164 2023-02-11 15:00:26,538 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=164, state=SUCCESS; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,41207,1676127604675 in 195 msec 2023-02-11 15:00:26,538 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN in 205 msec 2023-02-11 15:00:26,541 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN in 206 msec 2023-02-11 15:00:26,541 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=164, resume processing ppid=161 2023-02-11 15:00:26,541 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN in 206 msec 2023-02-11 15:00:26,542 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127626542"}]},"ts":"1676127626542"} 2023-02-11 15:00:26,543 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLED in hbase:meta 2023-02-11 15:00:26,545 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676127607597 to state=DISABLED 2023-02-11 15:00:26,546 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676127607597 in 225 msec 2023-02-11 15:00:26,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-11 15:00:26,575 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 161 completed 2023-02-11 15:00:26,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,588 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,589 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-11 15:00:26,596 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:26,596 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:26,596 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:26,596 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:26,596 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:26,596 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:26,598 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits] 2023-02-11 15:00:26,598 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits] 2023-02-11 15:00:26,598 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits] 2023-02-11 15:00:26,598 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits] 2023-02-11 15:00:26,598 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits] 2023-02-11 15:00:26,599 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits] 2023-02-11 15:00:26,602 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-830bf232363e40bbbca42f0cf5644c1d to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:26,602 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-2bab4cc59a7d4614827ba4ccb10f3bd6 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:26,603 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-80b2a4abbc984f83a8efd755415eaf73 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-80b2a4abbc984f83a8efd755415eaf73 2023-02-11 15:00:26,603 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-2a35e2af579a4f78aad755c9c1a42e33 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:26,603 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-46fbcfe2640b44ad951ffc998805ea49 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-46fbcfe2640b44ad951ffc998805ea49 2023-02-11 15:00:26,604 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-08c52f6c5115434fa8273a8c2162f5e1 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-08c52f6c5115434fa8273a8c2162f5e1 2023-02-11 15:00:26,604 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-a11a6bb05820487da1f0bf1a178b54e0 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-a11a6bb05820487da1f0bf1a178b54e0 2023-02-11 15:00:26,604 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-ea92c7cd9a554d4a849955a308ed7e13 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-ea92c7cd9a554d4a849955a308ed7e13 2023-02-11 15:00:26,605 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-4fefa1144af6408195c155495f67763b to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-4fefa1144af6408195c155495f67763b 2023-02-11 15:00:26,605 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-ef1ac041d7184ad1af9b3666b7d994ae to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:26,606 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-cf109e42ce0a4e698dd14c8965b4c03a to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:26,607 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-49a06ed7ffdc4dc9a2446e463bc2be02 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:26,609 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/27.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/27.seqid 2023-02-11 15:00:26,609 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/27.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/27.seqid 2023-02-11 15:00:26,609 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/27.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/27.seqid 2023-02-11 15:00:26,610 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:26,610 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:26,610 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/27.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/27.seqid 2023-02-11 15:00:26,610 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:26,611 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/27.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/27.seqid 2023-02-11 15:00:26,611 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:26,611 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:26,612 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/27.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/27.seqid 2023-02-11 15:00:26,612 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:26,612 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676127607597 regions 2023-02-11 15:00:26,613 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628 2023-02-11 15:00:26,613 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf] 2023-02-11 15:00:26,615 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc 2023-02-11 15:00:26,617 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba 2023-02-11 15:00:26,618 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 2023-02-11 15:00:26,619 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad 2023-02-11 15:00:26,620 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 2023-02-11 15:00:26,621 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 2023-02-11 15:00:26,622 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e 2023-02-11 15:00:26,624 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 2023-02-11 15:00:26,625 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 2023-02-11 15:00:26,626 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c 2023-02-11 15:00:26,627 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 2023-02-11 15:00:26,628 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 2023-02-11 15:00:26,628 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628 2023-02-11 15:00:26,630 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,653 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676127607597 from hbase:meta 2023-02-11 15:00:26,655 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676127607597' descriptor. 2023-02-11 15:00:26,656 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,656 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676127607597' from region states. 2023-02-11 15:00:26,656 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127626656"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:26,656 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127626656"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:26,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127626656"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:26,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127626656"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:26,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127626656"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:26,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127626656"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:26,659 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 15:00:26,659 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 15:00:26,659 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676127607597' as deleted. 2023-02-11 15:00:26,659 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676127626659"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:26,660 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676127607597 state from META 2023-02-11 15:00:26,662 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,663 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 in 82 msec 2023-02-11 15:00:26,684 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 15:00:26,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-11 15:00:26,841 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 174 completed 2023-02-11 15:00:26,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676127607597 snapshot=name: "snaptb1-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127616534 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 15:00:26,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676127607597 as table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 15:00:26,883 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/.tabledesc/.tableinfo.0000000001 2023-02-11 15:00:26,885 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127616534 type: DISABLED version: 2 ttl: 0 2023-02-11 15:00:26,885 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,886 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,888 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,891 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,893 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,895 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,898 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,900 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,902 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,904 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,906 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,908 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,909 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,911 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:26,911 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:26,911 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:26,911 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:26,912 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:26,912 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:26,912 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=78f87f7851c87dde38a4d94bc805db9b as 78f87f7851c87dde38a4d94bc805db9b in snapshot snaptb1-1676127607597 2023-02-11 15:00:26,912 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=94a0d6907c18dd6eadf539b9aa0231e2 as 94a0d6907c18dd6eadf539b9aa0231e2 in snapshot snaptb1-1676127607597 2023-02-11 15:00:26,912 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=7677bc32f6d3f7225f4cfdeb2068fc26 as 7677bc32f6d3f7225f4cfdeb2068fc26 in snapshot snaptb1-1676127607597 2023-02-11 15:00:26,912 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=704f07b3665f9baa8c5a5484e3bdbcd6 as 704f07b3665f9baa8c5a5484e3bdbcd6 in snapshot snaptb1-1676127607597 2023-02-11 15:00:26,912 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=cd2163f692eb299f43cb55be4d4dc49f as cd2163f692eb299f43cb55be4d4dc49f in snapshot snaptb1-1676127607597 2023-02-11 15:00:26,912 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=2c301b6be53a2e3d2c7174a30417ca8e as 2c301b6be53a2e3d2c7174a30417ca8e in snapshot snaptb1-1676127607597 2023-02-11 15:00:26,913 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:26,913 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:26,913 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:26,914 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:26,914 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:26,914 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676127607597', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:26,931 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:26,931 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 46fbcfe2640b44ad951ffc998805ea49 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,934 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:26,934 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 08c52f6c5115434fa8273a8c2162f5e1 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,934 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:26,934 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:26,935 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 830bf232363e40bbbca42f0cf5644c1d from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,935 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 2bab4cc59a7d4614827ba4ccb10f3bd6 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,935 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink cf109e42ce0a4e698dd14c8965b4c03a from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,937 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:26,937 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 80b2a4abbc984f83a8efd755415eaf73 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,938 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:26,938 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 2a35e2af579a4f78aad755c9c1a42e33 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,943 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 49a06ed7ffdc4dc9a2446e463bc2be02 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,947 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink ea92c7cd9a554d4a849955a308ed7e13 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,947 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink ef1ac041d7184ad1af9b3666b7d994ae from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,947 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 4fefa1144af6408195c155495f67763b from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,953 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a11a6bb05820487da1f0bf1a178b54e0 from cloned region in snapshot snaptb1-1676127607597 to table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:26,958 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing 2c301b6be53a2e3d2c7174a30417ca8e, disabling compactions & flushes 2023-02-11 15:00:26,958 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:26,958 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:26,958 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. after waiting 0 ms 2023-02-11 15:00:26,958 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:26,958 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:26,958 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for 2c301b6be53a2e3d2c7174a30417ca8e: Waiting for close lock at 1676127626958Disabling compacts and flushes for region at 1676127626958Disabling writes for close at 1676127626958Writing region close event to WAL at 1676127626958Closed at 1676127626958 2023-02-11 15:00:26,962 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing 7677bc32f6d3f7225f4cfdeb2068fc26, disabling compactions & flushes 2023-02-11 15:00:26,962 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:26,962 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing 94a0d6907c18dd6eadf539b9aa0231e2, disabling compactions & flushes 2023-02-11 15:00:26,962 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing 78f87f7851c87dde38a4d94bc805db9b, disabling compactions & flushes 2023-02-11 15:00:26,962 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:26,963 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:26,963 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. after waiting 0 ms 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. after waiting 0 ms 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. after waiting 0 ms 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing 704f07b3665f9baa8c5a5484e3bdbcd6, disabling compactions & flushes 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing cd2163f692eb299f43cb55be4d4dc49f, disabling compactions & flushes 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:26,963 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:26,963 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:26,963 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for 78f87f7851c87dde38a4d94bc805db9b: Waiting for close lock at 1676127626962Disabling compacts and flushes for region at 1676127626962Disabling writes for close at 1676127626963 (+1 ms)Writing region close event to WAL at 1676127626963Closed at 1676127626963 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:26,963 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. after waiting 0 ms 2023-02-11 15:00:26,963 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:26,964 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:26,964 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. after waiting 0 ms 2023-02-11 15:00:26,964 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:26,964 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:26,964 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:26,964 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for 94a0d6907c18dd6eadf539b9aa0231e2: Waiting for close lock at 1676127626962Disabling compacts and flushes for region at 1676127626962Disabling writes for close at 1676127626963 (+1 ms)Writing region close event to WAL at 1676127626964 (+1 ms)Closed at 1676127626964 2023-02-11 15:00:26,964 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:26,964 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Waiting for close lock at 1676127626962Disabling compacts and flushes for region at 1676127626962Disabling writes for close at 1676127626963 (+1 ms)Writing region close event to WAL at 1676127626963Closed at 1676127626963 2023-02-11 15:00:26,964 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for cd2163f692eb299f43cb55be4d4dc49f: Waiting for close lock at 1676127626963Disabling compacts and flushes for region at 1676127626963Disabling writes for close at 1676127626964 (+1 ms)Writing region close event to WAL at 1676127626964Closed at 1676127626964 2023-02-11 15:00:26,964 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Waiting for close lock at 1676127626963Disabling compacts and flushes for region at 1676127626963Disabling writes for close at 1676127626964 (+1 ms)Writing region close event to WAL at 1676127626964Closed at 1676127626964 2023-02-11 15:00:26,964 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127616534 type: DISABLED version: 2 ttl: 0 2023-02-11 15:00:26,965 INFO [PEWorker-2] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676127607597 on table=testRestoreSnapshot-1676127607597 completed! 2023-02-11 15:00:26,969 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127626969"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626969"}]},"ts":"1676127626969"} 2023-02-11 15:00:26,969 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626969"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626969"}]},"ts":"1676127626969"} 2023-02-11 15:00:26,969 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127626969"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626969"}]},"ts":"1676127626969"} 2023-02-11 15:00:26,969 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626969"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626969"}]},"ts":"1676127626969"} 2023-02-11 15:00:26,969 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626969"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626969"}]},"ts":"1676127626969"} 2023-02-11 15:00:26,969 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127626969"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127626969"}]},"ts":"1676127626969"} 2023-02-11 15:00:26,971 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 15:00:26,972 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127626972"}]},"ts":"1676127626972"} 2023-02-11 15:00:26,973 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLING in hbase:meta 2023-02-11 15:00:26,984 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 15:00:26,984 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 15:00:26,984 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 15:00:26,984 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 15:00:26,984 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 15:00:26,984 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN}] 2023-02-11 15:00:26,986 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN 2023-02-11 15:00:26,986 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN 2023-02-11 15:00:26,987 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN 2023-02-11 15:00:26,987 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN 2023-02-11 15:00:26,987 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=false 2023-02-11 15:00:26,988 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN 2023-02-11 15:00:26,988 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=false 2023-02-11 15:00:26,988 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=false 2023-02-11 15:00:26,988 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=false 2023-02-11 15:00:26,989 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN 2023-02-11 15:00:26,989 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=false 2023-02-11 15:00:26,990 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=false 2023-02-11 15:00:27,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 15:00:27,138 INFO [jenkins-hbase4:38517] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 15:00:27,141 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:27,141 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:27,141 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127627141"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127627141"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127627141"}]},"ts":"1676127627141"} 2023-02-11 15:00:27,141 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:27,141 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:27,141 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127627141"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127627141"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127627141"}]},"ts":"1676127627141"} 2023-02-11 15:00:27,141 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127627141"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127627141"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127627141"}]},"ts":"1676127627141"} 2023-02-11 15:00:27,141 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:27,141 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127627141"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127627141"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127627141"}]},"ts":"1676127627141"} 2023-02-11 15:00:27,141 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127627141"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127627141"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127627141"}]},"ts":"1676127627141"} 2023-02-11 15:00:27,142 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=179, state=RUNNABLE; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:27,145 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=177, state=RUNNABLE; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:27,146 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=176, state=RUNNABLE; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:27,146 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=180, state=RUNNABLE; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:27,147 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=178, state=RUNNABLE; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:27,147 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:27,148 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127627147"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127627147"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127627147"}]},"ts":"1676127627147"} 2023-02-11 15:00:27,149 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=181, state=RUNNABLE; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:27,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:27,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 15:00:27,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:27,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:27,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:27,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:27,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:27,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 15:00:27,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:27,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:27,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:27,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:27,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:27,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 15:00:27,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:27,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:27,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:27,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:27,302 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:27,302 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:27,303 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:27,303 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:27,303 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf 2023-02-11 15:00:27,303 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:27,303 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf 2023-02-11 15:00:27,303 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c301b6be53a2e3d2c7174a30417ca8e columnFamilyName cf 2023-02-11 15:00:27,304 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:27,304 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf 2023-02-11 15:00:27,304 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cd2163f692eb299f43cb55be4d4dc49f columnFamilyName cf 2023-02-11 15:00:27,304 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78f87f7851c87dde38a4d94bc805db9b columnFamilyName cf 2023-02-11 15:00:27,320 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-46fbcfe2640b44ad951ffc998805ea49 2023-02-11 15:00:27,320 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:27,320 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:27,326 DEBUG [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-ea92c7cd9a554d4a849955a308ed7e13 2023-02-11 15:00:27,326 INFO [StoreOpener-2c301b6be53a2e3d2c7174a30417ca8e-1] regionserver.HStore(310): Store=2c301b6be53a2e3d2c7174a30417ca8e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:27,327 DEBUG [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-4fefa1144af6408195c155495f67763b 2023-02-11 15:00:27,327 INFO [StoreOpener-cd2163f692eb299f43cb55be4d4dc49f-1] regionserver.HStore(310): Store=cd2163f692eb299f43cb55be4d4dc49f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:27,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:27,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:27,328 DEBUG [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:27,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:27,328 INFO [StoreOpener-78f87f7851c87dde38a4d94bc805db9b-1] regionserver.HStore(310): Store=78f87f7851c87dde38a4d94bc805db9b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:27,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:27,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:27,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:27,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:27,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 15:00:27,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:27,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2c301b6be53a2e3d2c7174a30417ca8e; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71018686, jitterRate=0.05826088786125183}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:27,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2c301b6be53a2e3d2c7174a30417ca8e: Running coprocessor pre-open hook at 1676127627300Writing region info on filesystem at 1676127627300Initializing all the Stores at 1676127627302 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127627302Cleaning up temporary data from old regions at 1676127627328 (+26 ms)Cleaning up detritus from prior splits at 1676127627329 (+1 ms)Running coprocessor post-open hooks at 1676127627333 (+4 ms)Region opened successfully at 1676127627333 2023-02-11 15:00:27,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e., pid=182, masterSystemTime=1676127627296 2023-02-11 15:00:27,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 15:00:27,335 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 78f87f7851c87dde38a4d94bc805db9b; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60390302, jitterRate=-0.1001143753528595}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:27,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:27,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 78f87f7851c87dde38a4d94bc805db9b: Running coprocessor pre-open hook at 1676127627302Writing region info on filesystem at 1676127627302Initializing all the Stores at 1676127627302Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127627303 (+1 ms)Cleaning up temporary data from old regions at 1676127627330 (+27 ms)Cleaning up detritus from prior splits at 1676127627330Running coprocessor post-open hooks at 1676127627335 (+5 ms)Region opened successfully at 1676127627335 2023-02-11 15:00:27,335 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:27,335 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:27,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 15:00:27,335 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:27,335 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127627335"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127627335"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127627335"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127627335"}]},"ts":"1676127627335"} 2023-02-11 15:00:27,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:27,335 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b., pid=187, masterSystemTime=1676127627299 2023-02-11 15:00:27,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:27,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:27,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:27,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:27,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:27,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:27,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 15:00:27,337 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:27,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:27,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:27,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:27,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:27,338 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:27,338 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:27,338 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127627338"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127627338"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127627338"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127627338"}]},"ts":"1676127627338"} 2023-02-11 15:00:27,338 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf 2023-02-11 15:00:27,339 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7677bc32f6d3f7225f4cfdeb2068fc26 columnFamilyName cf 2023-02-11 15:00:27,340 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:27,340 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=179 2023-02-11 15:00:27,341 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=179, state=SUCCESS; OpenRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,42505,1676127604785 in 195 msec 2023-02-11 15:00:27,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:27,341 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:27,341 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf 2023-02-11 15:00:27,342 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 704f07b3665f9baa8c5a5484e3bdbcd6 columnFamilyName cf 2023-02-11 15:00:27,342 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, ASSIGN in 356 msec 2023-02-11 15:00:27,344 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=181 2023-02-11 15:00:27,344 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=181, state=SUCCESS; OpenRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743 in 191 msec 2023-02-11 15:00:27,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 15:00:27,345 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened cd2163f692eb299f43cb55be4d4dc49f; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59170339, jitterRate=-0.11829324066638947}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:27,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for cd2163f692eb299f43cb55be4d4dc49f: Running coprocessor pre-open hook at 1676127627302Writing region info on filesystem at 1676127627302Initializing all the Stores at 1676127627302Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127627302Cleaning up temporary data from old regions at 1676127627329 (+27 ms)Cleaning up detritus from prior splits at 1676127627333 (+4 ms)Running coprocessor post-open hooks at 1676127627345 (+12 ms)Region opened successfully at 1676127627345 2023-02-11 15:00:27,345 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, ASSIGN in 360 msec 2023-02-11 15:00:27,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f., pid=183, masterSystemTime=1676127627297 2023-02-11 15:00:27,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:27,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:27,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:27,348 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-80b2a4abbc984f83a8efd755415eaf73 2023-02-11 15:00:27,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 15:00:27,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676127607597 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:27,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:27,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:27,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:27,349 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:27,350 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127627349"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127627349"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127627349"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127627349"}]},"ts":"1676127627349"} 2023-02-11 15:00:27,352 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:27,353 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:27,353 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf 2023-02-11 15:00:27,353 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94a0d6907c18dd6eadf539b9aa0231e2 columnFamilyName cf 2023-02-11 15:00:27,354 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:27,355 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=177 2023-02-11 15:00:27,355 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=177, state=SUCCESS; OpenRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,41207,1676127604675 in 210 msec 2023-02-11 15:00:27,356 DEBUG [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:27,356 INFO [StoreOpener-7677bc32f6d3f7225f4cfdeb2068fc26-1] regionserver.HStore(310): Store=7677bc32f6d3f7225f4cfdeb2068fc26/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:27,356 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, ASSIGN in 371 msec 2023-02-11 15:00:27,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:27,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:27,359 DEBUG [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-a11a6bb05820487da1f0bf1a178b54e0 2023-02-11 15:00:27,359 INFO [StoreOpener-704f07b3665f9baa8c5a5484e3bdbcd6-1] regionserver.HStore(310): Store=704f07b3665f9baa8c5a5484e3bdbcd6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:27,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:27,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:27,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:27,360 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-08c52f6c5115434fa8273a8c2162f5e1 2023-02-11 15:00:27,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:27,364 DEBUG [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:27,368 INFO [StoreOpener-94a0d6907c18dd6eadf539b9aa0231e2-1] regionserver.HStore(310): Store=94a0d6907c18dd6eadf539b9aa0231e2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:27,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 15:00:27,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 15:00:27,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:27,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7677bc32f6d3f7225f4cfdeb2068fc26; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71108082, jitterRate=0.059592992067337036}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:27,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 704f07b3665f9baa8c5a5484e3bdbcd6; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75017392, jitterRate=0.11784625053405762}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:27,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Running coprocessor pre-open hook at 1676127627336Writing region info on filesystem at 1676127627336Initializing all the Stores at 1676127627337 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127627337Cleaning up temporary data from old regions at 1676127627358 (+21 ms)Cleaning up detritus from prior splits at 1676127627358Running coprocessor post-open hooks at 1676127627369 (+11 ms)Region opened successfully at 1676127627369 2023-02-11 15:00:27,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Running coprocessor pre-open hook at 1676127627338Writing region info on filesystem at 1676127627338Initializing all the Stores at 1676127627339 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127627339Cleaning up temporary data from old regions at 1676127627360 (+21 ms)Cleaning up detritus from prior splits at 1676127627361 (+1 ms)Running coprocessor post-open hooks at 1676127627369 (+8 ms)Region opened successfully at 1676127627369 2023-02-11 15:00:27,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:27,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6., pid=186, masterSystemTime=1676127627299 2023-02-11 15:00:27,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26., pid=184, masterSystemTime=1676127627296 2023-02-11 15:00:27,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:27,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:27,371 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:27,371 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127627371"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127627371"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127627371"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127627371"}]},"ts":"1676127627371"} 2023-02-11 15:00:27,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:27,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:27,371 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:27,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:27,372 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127627371"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127627371"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127627371"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127627371"}]},"ts":"1676127627371"} 2023-02-11 15:00:27,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 15:00:27,374 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=176 2023-02-11 15:00:27,374 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 94a0d6907c18dd6eadf539b9aa0231e2; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63203805, jitterRate=-0.058189913630485535}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:27,374 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=176, state=SUCCESS; OpenRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785 in 226 msec 2023-02-11 15:00:27,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 94a0d6907c18dd6eadf539b9aa0231e2: Running coprocessor pre-open hook at 1676127627349Writing region info on filesystem at 1676127627349Initializing all the Stores at 1676127627350 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127627350Cleaning up temporary data from old regions at 1676127627369 (+19 ms)Cleaning up detritus from prior splits at 1676127627370 (+1 ms)Running coprocessor post-open hooks at 1676127627374 (+4 ms)Region opened successfully at 1676127627374 2023-02-11 15:00:27,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2., pid=185, masterSystemTime=1676127627297 2023-02-11 15:00:27,375 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=178 2023-02-11 15:00:27,375 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, ASSIGN in 390 msec 2023-02-11 15:00:27,375 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=178, state=SUCCESS; OpenRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743 in 226 msec 2023-02-11 15:00:27,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:27,376 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, ASSIGN in 391 msec 2023-02-11 15:00:27,376 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:27,376 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:27,376 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127627376"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127627376"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127627376"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127627376"}]},"ts":"1676127627376"} 2023-02-11 15:00:27,378 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=180 2023-02-11 15:00:27,378 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=180, state=SUCCESS; OpenRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,41207,1676127604675 in 231 msec 2023-02-11 15:00:27,380 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=180, resume processing ppid=175 2023-02-11 15:00:27,380 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, ASSIGN in 394 msec 2023-02-11 15:00:27,381 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127627381"}]},"ts":"1676127627381"} 2023-02-11 15:00:27,382 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=ENABLED in hbase:meta 2023-02-11 15:00:27,386 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676127607597 snapshot=name: "snaptb1-1676127607597" table: "testRestoreSnapshot-1676127607597" creation_time: 1676127616534 type: DISABLED version: 2 ttl: 0 ) in 522 msec 2023-02-11 15:00:27,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 15:00:27,623 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676127607597, procId: 175 completed 2023-02-11 15:00:27,624 DEBUG [Listener at localhost/39605] zookeeper.ReadOnlyZKClient(139): Connect 0x4ff51970 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:27,632 DEBUG [Listener at localhost/39605] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c9ce563, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:27,634 DEBUG [hconnection-0x69705b98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:27,637 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50688, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:27,640 DEBUG [hconnection-0x69705b98-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:27,643 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57064, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:27,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,802 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 15:00:27,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,966 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 15:00:27,968 DEBUG [hconnection-0x69705b98-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:27,969 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54250, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:27,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:27,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,195 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 15:00:28,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,371 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 15:00:28,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,522 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 15:00:28,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:28,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,179 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1550b85e to 127.0.0.1:58940 2023-02-11 15:00:29,179 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:29,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:29,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:30,362 INFO [Listener at localhost/39605] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676127607597 2023-02-11 15:00:30,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676127607597 2023-02-11 15:00:30,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:30,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-11 15:00:30,366 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127630366"}]},"ts":"1676127630366"} 2023-02-11 15:00:30,367 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLING in hbase:meta 2023-02-11 15:00:30,369 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676127607597 to state=DISABLING 2023-02-11 15:00:30,370 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN}] 2023-02-11 15:00:30,372 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN 2023-02-11 15:00:30,372 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN 2023-02-11 15:00:30,372 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN 2023-02-11 15:00:30,372 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN 2023-02-11 15:00:30,372 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN 2023-02-11 15:00:30,372 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:30,373 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127630372"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127630372"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127630372"}]},"ts":"1676127630372"} 2023-02-11 15:00:30,373 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:30,373 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:30,373 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127630373"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127630373"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127630373"}]},"ts":"1676127630373"} 2023-02-11 15:00:30,373 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127630373"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127630373"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127630373"}]},"ts":"1676127630373"} 2023-02-11 15:00:30,373 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:30,374 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127630373"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127630373"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127630373"}]},"ts":"1676127630373"} 2023-02-11 15:00:30,373 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:30,374 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=190, state=RUNNABLE; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:30,374 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127630373"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127630373"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127630373"}]},"ts":"1676127630373"} 2023-02-11 15:00:30,375 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=191, state=RUNNABLE; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:30,375 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=189, state=RUNNABLE; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:30,378 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=192, state=RUNNABLE; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:30,379 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=194, state=RUNNABLE; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:30,380 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN 2023-02-11 15:00:30,380 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:30,381 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127630380"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127630380"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127630380"}]},"ts":"1676127630380"} 2023-02-11 15:00:30,382 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=193, state=RUNNABLE; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:30,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:30,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:30,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:30,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 704f07b3665f9baa8c5a5484e3bdbcd6, disabling compactions & flushes 2023-02-11 15:00:30,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2c301b6be53a2e3d2c7174a30417ca8e, disabling compactions & flushes 2023-02-11 15:00:30,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing cd2163f692eb299f43cb55be4d4dc49f, disabling compactions & flushes 2023-02-11 15:00:30,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:30,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:30,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:30,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:30,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. after waiting 0 ms 2023-02-11 15:00:30,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. after waiting 0 ms 2023-02-11 15:00:30,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:30,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:30,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:30,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:30,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. after waiting 0 ms 2023-02-11 15:00:30,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:30,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 15:00:30,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 15:00:30,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 15:00:30,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f. 2023-02-11 15:00:30,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e. 2023-02-11 15:00:30,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for cd2163f692eb299f43cb55be4d4dc49f: Waiting for close lock at 1676127630530Running coprocessor pre-close hooks at 1676127630530Disabling compacts and flushes for region at 1676127630530Disabling writes for close at 1676127630532 (+2 ms)Writing region close event to WAL at 1676127630536 (+4 ms)Running coprocessor post-close hooks at 1676127630540 (+4 ms)Closed at 1676127630540 2023-02-11 15:00:30,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2c301b6be53a2e3d2c7174a30417ca8e: Waiting for close lock at 1676127630531Running coprocessor pre-close hooks at 1676127630531Disabling compacts and flushes for region at 1676127630531Disabling writes for close at 1676127630532 (+1 ms)Writing region close event to WAL at 1676127630537 (+5 ms)Running coprocessor post-close hooks at 1676127630540 (+3 ms)Closed at 1676127630540 2023-02-11 15:00:30,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6. 2023-02-11 15:00:30,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 704f07b3665f9baa8c5a5484e3bdbcd6: Waiting for close lock at 1676127630529Running coprocessor pre-close hooks at 1676127630529Disabling compacts and flushes for region at 1676127630529Disabling writes for close at 1676127630532 (+3 ms)Writing region close event to WAL at 1676127630537 (+5 ms)Running coprocessor post-close hooks at 1676127630540 (+3 ms)Closed at 1676127630540 2023-02-11 15:00:30,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:30,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:30,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 94a0d6907c18dd6eadf539b9aa0231e2, disabling compactions & flushes 2023-02-11 15:00:30,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:30,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:30,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. after waiting 0 ms 2023-02-11 15:00:30,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:30,543 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=cd2163f692eb299f43cb55be4d4dc49f, regionState=CLOSED 2023-02-11 15:00:30,547 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127630542"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630542"}]},"ts":"1676127630542"} 2023-02-11 15:00:30,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:30,549 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:30,549 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=2c301b6be53a2e3d2c7174a30417ca8e, regionState=CLOSED 2023-02-11 15:00:30,549 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127630549"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630549"}]},"ts":"1676127630549"} 2023-02-11 15:00:30,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7677bc32f6d3f7225f4cfdeb2068fc26, disabling compactions & flushes 2023-02-11 15:00:30,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:30,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:30,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:30,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:30,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. after waiting 0 ms 2023-02-11 15:00:30,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:30,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 78f87f7851c87dde38a4d94bc805db9b, disabling compactions & flushes 2023-02-11 15:00:30,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:30,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:30,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. after waiting 0 ms 2023-02-11 15:00:30,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:30,551 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=704f07b3665f9baa8c5a5484e3bdbcd6, regionState=CLOSED 2023-02-11 15:00:30,551 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127630551"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630551"}]},"ts":"1676127630551"} 2023-02-11 15:00:30,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 15:00:30,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2. 2023-02-11 15:00:30,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 94a0d6907c18dd6eadf539b9aa0231e2: Waiting for close lock at 1676127630542Running coprocessor pre-close hooks at 1676127630542Disabling compacts and flushes for region at 1676127630542Disabling writes for close at 1676127630542Writing region close event to WAL at 1676127630548 (+6 ms)Running coprocessor post-close hooks at 1676127630553 (+5 ms)Closed at 1676127630553 2023-02-11 15:00:30,554 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=190 2023-02-11 15:00:30,554 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=190, state=SUCCESS; CloseRegionProcedure cd2163f692eb299f43cb55be4d4dc49f, server=jenkins-hbase4.apache.org,41207,1676127604675 in 178 msec 2023-02-11 15:00:30,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 15:00:30,555 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=192 2023-02-11 15:00:30,555 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=192, state=SUCCESS; CloseRegionProcedure 2c301b6be53a2e3d2c7174a30417ca8e, server=jenkins-hbase4.apache.org,42505,1676127604785 in 176 msec 2023-02-11 15:00:30,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:30,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26. 2023-02-11 15:00:30,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7677bc32f6d3f7225f4cfdeb2068fc26: Waiting for close lock at 1676127630549Running coprocessor pre-close hooks at 1676127630549Disabling compacts and flushes for region at 1676127630549Disabling writes for close at 1676127630550 (+1 ms)Writing region close event to WAL at 1676127630551 (+1 ms)Running coprocessor post-close hooks at 1676127630555 (+4 ms)Closed at 1676127630555 2023-02-11 15:00:30,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 15:00:30,556 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=191 2023-02-11 15:00:30,556 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=94a0d6907c18dd6eadf539b9aa0231e2, regionState=CLOSED 2023-02-11 15:00:30,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=cd2163f692eb299f43cb55be4d4dc49f, UNASSIGN in 184 msec 2023-02-11 15:00:30,556 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676127630556"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630556"}]},"ts":"1676127630556"} 2023-02-11 15:00:30,556 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=191, state=SUCCESS; CloseRegionProcedure 704f07b3665f9baa8c5a5484e3bdbcd6, server=jenkins-hbase4.apache.org,33165,1676127604743 in 178 msec 2023-02-11 15:00:30,556 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b. 2023-02-11 15:00:30,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 78f87f7851c87dde38a4d94bc805db9b: Waiting for close lock at 1676127630550Running coprocessor pre-close hooks at 1676127630550Disabling compacts and flushes for region at 1676127630550Disabling writes for close at 1676127630550Writing region close event to WAL at 1676127630551 (+1 ms)Running coprocessor post-close hooks at 1676127630556 (+5 ms)Closed at 1676127630556 2023-02-11 15:00:30,556 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=2c301b6be53a2e3d2c7174a30417ca8e, UNASSIGN in 185 msec 2023-02-11 15:00:30,557 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=704f07b3665f9baa8c5a5484e3bdbcd6, UNASSIGN in 186 msec 2023-02-11 15:00:30,557 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:30,558 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=7677bc32f6d3f7225f4cfdeb2068fc26, regionState=CLOSED 2023-02-11 15:00:30,558 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127630558"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630558"}]},"ts":"1676127630558"} 2023-02-11 15:00:30,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:30,558 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=78f87f7851c87dde38a4d94bc805db9b, regionState=CLOSED 2023-02-11 15:00:30,558 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676127630558"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630558"}]},"ts":"1676127630558"} 2023-02-11 15:00:30,559 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=193 2023-02-11 15:00:30,559 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=193, state=SUCCESS; CloseRegionProcedure 94a0d6907c18dd6eadf539b9aa0231e2, server=jenkins-hbase4.apache.org,41207,1676127604675 in 175 msec 2023-02-11 15:00:30,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=94a0d6907c18dd6eadf539b9aa0231e2, UNASSIGN in 189 msec 2023-02-11 15:00:30,560 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=189 2023-02-11 15:00:30,560 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=189, state=SUCCESS; CloseRegionProcedure 7677bc32f6d3f7225f4cfdeb2068fc26, server=jenkins-hbase4.apache.org,42505,1676127604785 in 184 msec 2023-02-11 15:00:30,561 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=194 2023-02-11 15:00:30,561 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=194, state=SUCCESS; CloseRegionProcedure 78f87f7851c87dde38a4d94bc805db9b, server=jenkins-hbase4.apache.org,33165,1676127604743 in 180 msec 2023-02-11 15:00:30,561 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=7677bc32f6d3f7225f4cfdeb2068fc26, UNASSIGN in 190 msec 2023-02-11 15:00:30,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=194, resume processing ppid=188 2023-02-11 15:00:30,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676127607597, region=78f87f7851c87dde38a4d94bc805db9b, UNASSIGN in 191 msec 2023-02-11 15:00:30,563 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127630562"}]},"ts":"1676127630562"} 2023-02-11 15:00:30,563 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676127607597, state=DISABLED in hbase:meta 2023-02-11 15:00:30,565 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676127607597 to state=DISABLED 2023-02-11 15:00:30,566 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676127607597 in 202 msec 2023-02-11 15:00:30,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-11 15:00:30,617 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 188 completed 2023-02-11 15:00:30,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676127607597 2023-02-11 15:00:30,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:30,620 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:30,621 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:30,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-11 15:00:30,627 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:30,627 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:30,627 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:30,627 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:30,627 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:30,627 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:30,629 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits] 2023-02-11 15:00:30,629 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits] 2023-02-11 15:00:30,629 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits] 2023-02-11 15:00:30,629 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits] 2023-02-11 15:00:30,629 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits] 2023-02-11 15:00:30,630 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits] 2023-02-11 15:00:30,633 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-80b2a4abbc984f83a8efd755415eaf73 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-80b2a4abbc984f83a8efd755415eaf73. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626601 currentMtime: 1676127626947 2023-02-11 15:00:30,633 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-08c52f6c5115434fa8273a8c2162f5e1 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-08c52f6c5115434fa8273a8c2162f5e1. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626603 currentMtime: 1676127626941 2023-02-11 15:00:30,633 ERROR [HFileArchiver-8] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-2bab4cc59a7d4614827ba4ccb10f3bd6 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-2bab4cc59a7d4614827ba4ccb10f3bd6. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626601 currentMtime: 1676127626946 2023-02-11 15:00:30,634 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-2a35e2af579a4f78aad755c9c1a42e33 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-2a35e2af579a4f78aad755c9c1a42e33. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626602 currentMtime: 1676127626947 2023-02-11 15:00:30,634 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-80b2a4abbc984f83a8efd755415eaf73 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-80b2a4abbc984f83a8efd755415eaf73.1676127630629. 2023-02-11 15:00:30,634 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-08c52f6c5115434fa8273a8c2162f5e1 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-08c52f6c5115434fa8273a8c2162f5e1.1676127630629. 2023-02-11 15:00:30,634 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-830bf232363e40bbbca42f0cf5644c1d already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-830bf232363e40bbbca42f0cf5644c1d. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626601 currentMtime: 1676127626944 2023-02-11 15:00:30,634 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-46fbcfe2640b44ad951ffc998805ea49 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-46fbcfe2640b44ad951ffc998805ea49. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626602 currentMtime: 1676127626935 2023-02-11 15:00:30,634 INFO [HFileArchiver-8] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-2bab4cc59a7d4614827ba4ccb10f3bd6 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-2bab4cc59a7d4614827ba4ccb10f3bd6.1676127630630. 2023-02-11 15:00:30,634 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-2a35e2af579a4f78aad755c9c1a42e33 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-2a35e2af579a4f78aad755c9c1a42e33.1676127630630. 2023-02-11 15:00:30,635 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-830bf232363e40bbbca42f0cf5644c1d to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-830bf232363e40bbbca42f0cf5644c1d.1676127630630. 2023-02-11 15:00:30,635 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-80b2a4abbc984f83a8efd755415eaf73 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-80b2a4abbc984f83a8efd755415eaf73 2023-02-11 15:00:30,635 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-08c52f6c5115434fa8273a8c2162f5e1 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-08c52f6c5115434fa8273a8c2162f5e1 2023-02-11 15:00:30,635 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-46fbcfe2640b44ad951ffc998805ea49 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-46fbcfe2640b44ad951ffc998805ea49.1676127630629. 2023-02-11 15:00:30,635 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-830bf232363e40bbbca42f0cf5644c1d to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-830bf232363e40bbbca42f0cf5644c1d 2023-02-11 15:00:30,636 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-2a35e2af579a4f78aad755c9c1a42e33 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-2a35e2af579a4f78aad755c9c1a42e33 2023-02-11 15:00:30,636 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-2bab4cc59a7d4614827ba4ccb10f3bd6 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-2bab4cc59a7d4614827ba4ccb10f3bd6 2023-02-11 15:00:30,637 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-46fbcfe2640b44ad951ffc998805ea49 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-46fbcfe2640b44ad951ffc998805ea49 2023-02-11 15:00:30,637 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-49a06ed7ffdc4dc9a2446e463bc2be02 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-49a06ed7ffdc4dc9a2446e463bc2be02. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626605 currentMtime: 1676127626960 2023-02-11 15:00:30,638 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-a11a6bb05820487da1f0bf1a178b54e0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-a11a6bb05820487da1f0bf1a178b54e0. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626603 currentMtime: 1676127626963 2023-02-11 15:00:30,638 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-ef1ac041d7184ad1af9b3666b7d994ae already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-ef1ac041d7184ad1af9b3666b7d994ae. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626604 currentMtime: 1676127626953 2023-02-11 15:00:30,638 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-4fefa1144af6408195c155495f67763b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-4fefa1144af6408195c155495f67763b. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626604 currentMtime: 1676127626963 2023-02-11 15:00:30,638 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-49a06ed7ffdc4dc9a2446e463bc2be02 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-49a06ed7ffdc4dc9a2446e463bc2be02.1676127630629. 2023-02-11 15:00:30,638 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-a11a6bb05820487da1f0bf1a178b54e0 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-a11a6bb05820487da1f0bf1a178b54e0.1676127630630. 2023-02-11 15:00:30,638 ERROR [HFileArchiver-8] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-ea92c7cd9a554d4a849955a308ed7e13 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-ea92c7cd9a554d4a849955a308ed7e13. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626603 currentMtime: 1676127626956 2023-02-11 15:00:30,638 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-ef1ac041d7184ad1af9b3666b7d994ae to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-ef1ac041d7184ad1af9b3666b7d994ae.1676127630629. 2023-02-11 15:00:30,639 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-4fefa1144af6408195c155495f67763b to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-4fefa1144af6408195c155495f67763b.1676127630630. 2023-02-11 15:00:30,639 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-cf109e42ce0a4e698dd14c8965b4c03a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-cf109e42ce0a4e698dd14c8965b4c03a. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626604 currentMtime: 1676127626958 2023-02-11 15:00:30,639 INFO [HFileArchiver-8] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-ea92c7cd9a554d4a849955a308ed7e13 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-ea92c7cd9a554d4a849955a308ed7e13.1676127630630. 2023-02-11 15:00:30,639 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-49a06ed7ffdc4dc9a2446e463bc2be02 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/cf/testRestoreSnapshot-1676127607597=94a0d6907c18dd6eadf539b9aa0231e2-49a06ed7ffdc4dc9a2446e463bc2be02 2023-02-11 15:00:30,639 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-a11a6bb05820487da1f0bf1a178b54e0 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/cf/testRestoreSnapshot-1676127607597=704f07b3665f9baa8c5a5484e3bdbcd6-a11a6bb05820487da1f0bf1a178b54e0 2023-02-11 15:00:30,640 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-cf109e42ce0a4e698dd14c8965b4c03a to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-cf109e42ce0a4e698dd14c8965b4c03a.1676127630629. 2023-02-11 15:00:30,640 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-ef1ac041d7184ad1af9b3666b7d994ae to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/cf/testRestoreSnapshot-1676127607597=7677bc32f6d3f7225f4cfdeb2068fc26-ef1ac041d7184ad1af9b3666b7d994ae 2023-02-11 15:00:30,640 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-4fefa1144af6408195c155495f67763b to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/cf/testRestoreSnapshot-1676127607597=cd2163f692eb299f43cb55be4d4dc49f-4fefa1144af6408195c155495f67763b 2023-02-11 15:00:30,640 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-ea92c7cd9a554d4a849955a308ed7e13 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/cf/testRestoreSnapshot-1676127607597=2c301b6be53a2e3d2c7174a30417ca8e-ea92c7cd9a554d4a849955a308ed7e13 2023-02-11 15:00:30,641 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-cf109e42ce0a4e698dd14c8965b4c03a to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/cf/testRestoreSnapshot-1676127607597=78f87f7851c87dde38a4d94bc805db9b-cf109e42ce0a4e698dd14c8965b4c03a 2023-02-11 15:00:30,643 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26/recovered.edits/18.seqid 2023-02-11 15:00:30,643 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6/recovered.edits/18.seqid 2023-02-11 15:00:30,643 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2/recovered.edits/18.seqid 2023-02-11 15:00:30,644 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/7677bc32f6d3f7225f4cfdeb2068fc26 2023-02-11 15:00:30,644 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/704f07b3665f9baa8c5a5484e3bdbcd6 2023-02-11 15:00:30,644 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f/recovered.edits/18.seqid 2023-02-11 15:00:30,644 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/94a0d6907c18dd6eadf539b9aa0231e2 2023-02-11 15:00:30,645 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/cd2163f692eb299f43cb55be4d4dc49f 2023-02-11 15:00:30,645 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b/recovered.edits/18.seqid 2023-02-11 15:00:30,645 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e/recovered.edits/18.seqid 2023-02-11 15:00:30,646 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/78f87f7851c87dde38a4d94bc805db9b 2023-02-11 15:00:30,646 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testRestoreSnapshot-1676127607597/2c301b6be53a2e3d2c7174a30417ca8e 2023-02-11 15:00:30,646 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676127607597 regions 2023-02-11 15:00:30,646 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628 2023-02-11 15:00:30,647 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf] 2023-02-11 15:00:30,648 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626615 currentMtime: 1676127626888 2023-02-11 15:00:30,649 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc.1676127630647. 2023-02-11 15:00:30,649 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211551717f610a3478e9b403fb819d6f2cc 2023-02-11 15:00:30,650 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626616 currentMtime: 1676127626890 2023-02-11 15:00:30,651 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba.1676127630647. 2023-02-11 15:00:30,651 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-a87ff679a2f3e71d9181a67b7542122c20230211edcd5ddd1d3d4877a03f7b832f4abbba 2023-02-11 15:00:30,652 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626617 currentMtime: 1676127626893 2023-02-11 15:00:30,653 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06.1676127630647. 2023-02-11 15:00:30,654 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b2023021196d4f0e41cf8492baa0977cc8d35be06 2023-02-11 15:00:30,654 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626618 currentMtime: 1676127626895 2023-02-11 15:00:30,655 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad.1676127630647. 2023-02-11 15:00:30,655 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c4ca4238a0b923820dcc509a6f75849b20230211980bc430d6c34b1bbe2a6878805d1aad 2023-02-11 15:00:30,656 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626619 currentMtime: 1676127626898 2023-02-11 15:00:30,657 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641.1676127630647. 2023-02-11 15:00:30,657 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c20230211092390e5558c49b684b2fd7cf5412641 2023-02-11 15:00:30,658 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626621 currentMtime: 1676127626900 2023-02-11 15:00:30,658 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62.1676127630647. 2023-02-11 15:00:30,659 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-c81e728d9d4c2f636f067f89cc14862c202302117ee210963c0a406293feb032bac21b62 2023-02-11 15:00:30,660 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626622 currentMtime: 1676127626902 2023-02-11 15:00:30,660 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e.1676127630647. 2023-02-11 15:00:30,661 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e2023021196b3dac664dd47aa83fcbba0cf14d96e 2023-02-11 15:00:30,662 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626623 currentMtime: 1676127626904 2023-02-11 15:00:30,662 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0.1676127630647. 2023-02-11 15:00:30,663 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-d41d8cd98f00b204e9800998ecf8427e20230211c5a2aa47137c4277967d8e50a1c9b1c0 2023-02-11 15:00:30,664 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626624 currentMtime: 1676127626906 2023-02-11 15:00:30,664 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130.1676127630647. 2023-02-11 15:00:30,665 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302114de23dd4644041348d22d49f3e95a130 2023-02-11 15:00:30,665 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626625 currentMtime: 1676127626907 2023-02-11 15:00:30,666 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c.1676127630647. 2023-02-11 15:00:30,666 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-e4da3b7fbbce2345d7772b0674a318d5202302118db65968f2ea4021b70764a099d5f77c 2023-02-11 15:00:30,667 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626626 currentMtime: 1676127626909 2023-02-11 15:00:30,667 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557.1676127630647. 2023-02-11 15:00:30,668 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf32023021109fed36033034f33a97507101c90d557 2023-02-11 15:00:30,669 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156. archiveLen: 0 currentLen: 0 archiveMtime: 1676127626628 currentMtime: 1676127626911 2023-02-11 15:00:30,669 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156.1676127630647. 2023-02-11 15:00:30,670 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628/cf/testRestoreSnapshot-1676127607597=be8dfc6e3907aa03a0317253d205c628-eccbc87e4b5ce2fe28308fd9f2a7baf3202302110f50345ff26f4f10be411fca61426156 2023-02-11 15:00:30,670 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testRestoreSnapshot-1676127607597/be8dfc6e3907aa03a0317253d205c628 2023-02-11 15:00:30,671 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:30,674 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676127607597 from hbase:meta 2023-02-11 15:00:30,676 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676127607597' descriptor. 2023-02-11 15:00:30,677 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:30,677 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676127607597' from region states. 2023-02-11 15:00:30,677 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127630677"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:30,677 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127630677"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:30,677 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127630677"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:30,677 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127630677"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:30,677 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127630677"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:30,677 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127630677"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:30,679 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 15:00:30,679 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7677bc32f6d3f7225f4cfdeb2068fc26, NAME => 'testRestoreSnapshot-1676127607597,,1676127607655.7677bc32f6d3f7225f4cfdeb2068fc26.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => cd2163f692eb299f43cb55be4d4dc49f, NAME => 'testRestoreSnapshot-1676127607597,1,1676127607655.cd2163f692eb299f43cb55be4d4dc49f.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 704f07b3665f9baa8c5a5484e3bdbcd6, NAME => 'testRestoreSnapshot-1676127607597,2,1676127607655.704f07b3665f9baa8c5a5484e3bdbcd6.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 2c301b6be53a2e3d2c7174a30417ca8e, NAME => 'testRestoreSnapshot-1676127607597,3,1676127607655.2c301b6be53a2e3d2c7174a30417ca8e.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 94a0d6907c18dd6eadf539b9aa0231e2, NAME => 'testRestoreSnapshot-1676127607597,4,1676127607655.94a0d6907c18dd6eadf539b9aa0231e2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 78f87f7851c87dde38a4d94bc805db9b, NAME => 'testRestoreSnapshot-1676127607597,5,1676127607655.78f87f7851c87dde38a4d94bc805db9b.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 15:00:30,679 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676127607597' as deleted. 2023-02-11 15:00:30,679 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676127607597","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676127630679"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:30,680 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676127607597 state from META 2023-02-11 15:00:30,682 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 2023-02-11 15:00:30,683 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676127607597 in 64 msec 2023-02-11 15:00:30,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-11 15:00:30,872 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676127607597, procId: 201 completed 2023-02-11 15:00:30,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676127607597" 2023-02-11 15:00:30,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676127607597 2023-02-11 15:00:30,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676127607597" 2023-02-11 15:00:30,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676127607597 2023-02-11 15:00:30,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1676127607597" 2023-02-11 15:00:30,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676127607597 2023-02-11 15:00:30,910 INFO [Listener at localhost/39605] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=426 (was 359) Potentially hanging thread: hconnection-0x7b9b1635-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_368218882_15 at /127.0.0.1:57090 [Waiting for operation #474] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1091039350_15 at /127.0.0.1:47748 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_368218882_15 at /127.0.0.1:52654 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1091039350_15 at /127.0.0.1:52674 [Waiting for operation #517] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58940@0x4ff51970-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1787490620_15 at /127.0.0.1:41116 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1501744286_15 at /127.0.0.1:57126 [Waiting for operation #513] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4553a206-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1501744286_15 at /127.0.0.1:41110 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1091039350_15 at /127.0.0.1:41122 [Waiting for operation #507] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4553a206-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4553a206-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58940@0x4ff51970-SendThread(127.0.0.1:58940) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_368218882_15 at /127.0.0.1:41120 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_368218882_15 at /127.0.0.1:57094 [Waiting for operation #480] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1091039350_15 at /127.0.0.1:41138 [Waiting for operation #535] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1501744286_15 at /127.0.0.1:52652 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1787490620_15 at /127.0.0.1:52670 [Waiting for operation #504] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-metaLookup-shared--pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1501744286_15 at /127.0.0.1:41100 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58940@0x4ff51970 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/663225396.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1787490620_15 at /127.0.0.1:52602 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1787490620_15 at /127.0.0.1:41108 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4553a206-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=812 (was 691) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=283 (was 259) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 171), AvailableMemoryMB=3243 (was 3688) 2023-02-11 15:00:30,926 INFO [Listener at localhost/39605] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=426, OpenFileDescriptor=812, MaxFileDescriptor=60000, SystemLoadAverage=283, ProcessCount=168, AvailableMemoryMB=3243 2023-02-11 15:00:30,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1676127630926', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-11 15:00:30,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:30,931 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676127630926 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 15:00:30,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676127630926" procId is: 202 2023-02-11 15:00:30,931 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676127630926 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 15:00:30,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 15:00:30,935 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:30,935 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:30,935 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:30,935 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:30,936 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:30,935 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:30,936 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 empty. 2023-02-11 15:00:30,936 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 empty. 2023-02-11 15:00:30,936 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c empty. 2023-02-11 15:00:30,936 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 empty. 2023-02-11 15:00:30,936 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:30,936 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 empty. 2023-02-11 15:00:30,936 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d empty. 2023-02-11 15:00:30,937 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:30,937 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:30,937 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:30,938 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:30,938 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:30,938 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676127630926 regions 2023-02-11 15:00:30,951 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/.tabledesc/.tableinfo.0000000001 2023-02-11 15:00:30,952 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(7671): creating {ENCODED => a25d52f50b583b8024a28335f497bb9c, NAME => 'testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676127630926', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:30,952 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(7671): creating {ENCODED => e7f1f013dd1caf6b5254f795ef0a3ed7, NAME => 'testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676127630926', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:30,952 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(7671): creating {ENCODED => 238478a8a495efd2d422ce3a8631bbc2, NAME => 'testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676127630926', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:30,975 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:30,975 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1603): Closing 238478a8a495efd2d422ce3a8631bbc2, disabling compactions & flushes 2023-02-11 15:00:30,975 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:30,975 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:30,975 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. after waiting 0 ms 2023-02-11 15:00:30,975 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:30,975 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:30,975 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1557): Region close journal for 238478a8a495efd2d422ce3a8631bbc2: Waiting for close lock at 1676127630975Disabling compacts and flushes for region at 1676127630975Disabling writes for close at 1676127630975Writing region close event to WAL at 1676127630975Closed at 1676127630975 2023-02-11 15:00:30,976 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(7671): creating {ENCODED => 43b897ff4788f8ff736d623d593d53f4, NAME => 'testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676127630926', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1603): Closing a25d52f50b583b8024a28335f497bb9c, disabling compactions & flushes 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1603): Closing e7f1f013dd1caf6b5254f795ef0a3ed7, disabling compactions & flushes 2023-02-11 15:00:30,980 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:30,980 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. after waiting 0 ms 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. after waiting 0 ms 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:30,980 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:30,980 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1557): Region close journal for a25d52f50b583b8024a28335f497bb9c: Waiting for close lock at 1676127630980Disabling compacts and flushes for region at 1676127630980Disabling writes for close at 1676127630980Writing region close event to WAL at 1676127630980Closed at 1676127630980 2023-02-11 15:00:30,980 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1557): Region close journal for e7f1f013dd1caf6b5254f795ef0a3ed7: Waiting for close lock at 1676127630980Disabling compacts and flushes for region at 1676127630980Disabling writes for close at 1676127630980Writing region close event to WAL at 1676127630980Closed at 1676127630980 2023-02-11 15:00:30,981 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(7671): creating {ENCODED => 08d6264e7bb30445380a92d3761a3474, NAME => 'testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676127630926', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:30,981 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(7671): creating {ENCODED => 8de7ceba3e95ef9d71ef663e0063791d, NAME => 'testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676127630926', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp 2023-02-11 15:00:30,993 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:30,993 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1603): Closing 08d6264e7bb30445380a92d3761a3474, disabling compactions & flushes 2023-02-11 15:00:30,993 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:30,993 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:30,993 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. after waiting 0 ms 2023-02-11 15:00:30,993 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:30,993 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:30,993 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-2] regionserver.HRegion(1557): Region close journal for 08d6264e7bb30445380a92d3761a3474: Waiting for close lock at 1676127630993Disabling compacts and flushes for region at 1676127630993Disabling writes for close at 1676127630993Writing region close event to WAL at 1676127630993Closed at 1676127630993 2023-02-11 15:00:30,993 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:30,993 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1603): Closing 43b897ff4788f8ff736d623d593d53f4, disabling compactions & flushes 2023-02-11 15:00:30,993 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:30,994 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:30,994 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. after waiting 0 ms 2023-02-11 15:00:30,994 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:30,994 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:30,994 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-0] regionserver.HRegion(1557): Region close journal for 43b897ff4788f8ff736d623d593d53f4: Waiting for close lock at 1676127630993Disabling compacts and flushes for region at 1676127630993Disabling writes for close at 1676127630994 (+1 ms)Writing region close event to WAL at 1676127630994Closed at 1676127630994 2023-02-11 15:00:30,995 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:30,995 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1603): Closing 8de7ceba3e95ef9d71ef663e0063791d, disabling compactions & flushes 2023-02-11 15:00:30,995 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:30,995 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:30,995 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. after waiting 0 ms 2023-02-11 15:00:30,995 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:30,995 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:30,995 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676127630926-pool-1] regionserver.HRegion(1557): Region close journal for 8de7ceba3e95ef9d71ef663e0063791d: Waiting for close lock at 1676127630995Disabling compacts and flushes for region at 1676127630995Disabling writes for close at 1676127630995Writing region close event to WAL at 1676127630995Closed at 1676127630995 2023-02-11 15:00:30,997 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676127630926 execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 15:00:30,998 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127630998"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630998"}]},"ts":"1676127630998"} 2023-02-11 15:00:30,998 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127630998"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630998"}]},"ts":"1676127630998"} 2023-02-11 15:00:30,998 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127630998"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630998"}]},"ts":"1676127630998"} 2023-02-11 15:00:30,998 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127630998"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630998"}]},"ts":"1676127630998"} 2023-02-11 15:00:30,998 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127630998"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630998"}]},"ts":"1676127630998"} 2023-02-11 15:00:30,999 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127630998"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127630998"}]},"ts":"1676127630998"} 2023-02-11 15:00:31,003 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 15:00:31,003 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676127630926 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 15:00:31,003 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127631003"}]},"ts":"1676127631003"} 2023-02-11 15:00:31,004 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=ENABLING in hbase:meta 2023-02-11 15:00:31,010 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 15:00:31,012 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 15:00:31,012 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 15:00:31,012 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 15:00:31,012 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 15:00:31,012 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN}] 2023-02-11 15:00:31,014 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN 2023-02-11 15:00:31,014 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN 2023-02-11 15:00:31,014 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN 2023-02-11 15:00:31,014 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN 2023-02-11 15:00:31,014 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN 2023-02-11 15:00:31,015 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=false 2023-02-11 15:00:31,015 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=false 2023-02-11 15:00:31,015 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=false 2023-02-11 15:00:31,015 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=false 2023-02-11 15:00:31,016 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=false 2023-02-11 15:00:31,016 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN 2023-02-11 15:00:31,017 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=false 2023-02-11 15:00:31,165 INFO [jenkins-hbase4:38517] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 15:00:31,168 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:31,168 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:31,168 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631168"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631168"}]},"ts":"1676127631168"} 2023-02-11 15:00:31,168 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:31,169 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127631168"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631168"}]},"ts":"1676127631168"} 2023-02-11 15:00:31,169 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631168"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631168"}]},"ts":"1676127631168"} 2023-02-11 15:00:31,168 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:31,168 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:31,169 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127631168"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631168"}]},"ts":"1676127631168"} 2023-02-11 15:00:31,169 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631168"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631168"}]},"ts":"1676127631168"} 2023-02-11 15:00:31,170 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=207, state=RUNNABLE; OpenRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:31,170 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=208, state=RUNNABLE; OpenRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:31,171 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=205, state=RUNNABLE; OpenRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:31,172 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=203, state=RUNNABLE; OpenRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:31,172 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=206, state=RUNNABLE; OpenRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:31,174 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:31,174 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631174"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631174"}]},"ts":"1676127631174"} 2023-02-11 15:00:31,175 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=204, state=RUNNABLE; OpenRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:31,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 15:00:31,324 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:31,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 238478a8a495efd2d422ce3a8631bbc2, NAME => 'testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 15:00:31,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:31,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:31,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:31,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:31,325 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:31,325 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:31,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a25d52f50b583b8024a28335f497bb9c, NAME => 'testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 15:00:31,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:31,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:31,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:31,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:31,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8de7ceba3e95ef9d71ef663e0063791d, NAME => 'testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 15:00:31,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:31,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:31,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:31,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:31,326 INFO [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:31,327 INFO [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:31,327 INFO [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:31,328 DEBUG [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf 2023-02-11 15:00:31,328 DEBUG [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf 2023-02-11 15:00:31,328 DEBUG [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf 2023-02-11 15:00:31,328 DEBUG [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf 2023-02-11 15:00:31,328 INFO [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 238478a8a495efd2d422ce3a8631bbc2 columnFamilyName cf 2023-02-11 15:00:31,328 INFO [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a25d52f50b583b8024a28335f497bb9c columnFamilyName cf 2023-02-11 15:00:31,328 DEBUG [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf 2023-02-11 15:00:31,328 DEBUG [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf 2023-02-11 15:00:31,329 INFO [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8de7ceba3e95ef9d71ef663e0063791d columnFamilyName cf 2023-02-11 15:00:31,329 INFO [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] regionserver.HStore(310): Store=238478a8a495efd2d422ce3a8631bbc2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:31,329 INFO [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] regionserver.HStore(310): Store=a25d52f50b583b8024a28335f497bb9c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:31,329 INFO [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] regionserver.HStore(310): Store=8de7ceba3e95ef9d71ef663e0063791d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:31,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:31,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:31,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:31,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:31,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:31,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:31,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:31,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:31,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:31,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:31,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:31,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 238478a8a495efd2d422ce3a8631bbc2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64068861, jitterRate=-0.04529957473278046}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:31,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a25d52f50b583b8024a28335f497bb9c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72623377, jitterRate=0.08217264711856842}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:31,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 238478a8a495efd2d422ce3a8631bbc2: Running coprocessor pre-open hook at 1676127631325Writing region info on filesystem at 1676127631325Initializing all the Stores at 1676127631326 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127631326Cleaning up temporary data from old regions at 1676127631331 (+5 ms)Cleaning up detritus from prior splits at 1676127631331Running coprocessor post-open hooks at 1676127631337 (+6 ms)Region opened successfully at 1676127631337 2023-02-11 15:00:31,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a25d52f50b583b8024a28335f497bb9c: Running coprocessor pre-open hook at 1676127631326Writing region info on filesystem at 1676127631326Initializing all the Stores at 1676127631326Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127631326Cleaning up temporary data from old regions at 1676127631331 (+5 ms)Cleaning up detritus from prior splits at 1676127631331Running coprocessor post-open hooks at 1676127631337 (+6 ms)Region opened successfully at 1676127631337 2023-02-11 15:00:31,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:31,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8de7ceba3e95ef9d71ef663e0063791d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69839404, jitterRate=0.04068821668624878}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:31,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8de7ceba3e95ef9d71ef663e0063791d: Running coprocessor pre-open hook at 1676127631326Writing region info on filesystem at 1676127631326Initializing all the Stores at 1676127631327 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127631327Cleaning up temporary data from old regions at 1676127631331 (+4 ms)Cleaning up detritus from prior splits at 1676127631332 (+1 ms)Running coprocessor post-open hooks at 1676127631338 (+6 ms)Region opened successfully at 1676127631338 2023-02-11 15:00:31,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c., pid=211, masterSystemTime=1676127631323 2023-02-11 15:00:31,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2., pid=212, masterSystemTime=1676127631322 2023-02-11 15:00:31,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d., pid=210, masterSystemTime=1676127631323 2023-02-11 15:00:31,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:31,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:31,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:31,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 43b897ff4788f8ff736d623d593d53f4, NAME => 'testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 15:00:31,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:31,340 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:31,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:31,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:31,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:31,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:31,340 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631340"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127631340"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127631340"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127631340"}]},"ts":"1676127631340"} 2023-02-11 15:00:31,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:31,341 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:31,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:31,341 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127631340"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127631340"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127631340"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127631340"}]},"ts":"1676127631340"} 2023-02-11 15:00:31,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 08d6264e7bb30445380a92d3761a3474, NAME => 'testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 15:00:31,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:31,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:31,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:31,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:31,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e7f1f013dd1caf6b5254f795ef0a3ed7, NAME => 'testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 15:00:31,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:31,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:31,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:31,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:31,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:31,342 INFO [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:31,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:31,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:31,342 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:31,342 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127631342"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127631342"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127631342"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127631342"}]},"ts":"1676127631342"} 2023-02-11 15:00:31,343 INFO [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:31,344 INFO [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:31,344 DEBUG [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf 2023-02-11 15:00:31,344 DEBUG [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf 2023-02-11 15:00:31,345 INFO [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 43b897ff4788f8ff736d623d593d53f4 columnFamilyName cf 2023-02-11 15:00:31,345 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=205 2023-02-11 15:00:31,345 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=205, state=SUCCESS; OpenRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785 in 171 msec 2023-02-11 15:00:31,345 DEBUG [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf 2023-02-11 15:00:31,345 DEBUG [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf 2023-02-11 15:00:31,345 INFO [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7f1f013dd1caf6b5254f795ef0a3ed7 columnFamilyName cf 2023-02-11 15:00:31,345 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=203 2023-02-11 15:00:31,346 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=203, state=SUCCESS; OpenRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675 in 171 msec 2023-02-11 15:00:31,346 DEBUG [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf 2023-02-11 15:00:31,346 DEBUG [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf 2023-02-11 15:00:31,346 INFO [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] regionserver.HStore(310): Store=43b897ff4788f8ff736d623d593d53f4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:31,346 INFO [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08d6264e7bb30445380a92d3761a3474 columnFamilyName cf 2023-02-11 15:00:31,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN in 333 msec 2023-02-11 15:00:31,347 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=208 2023-02-11 15:00:31,347 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=208, state=SUCCESS; OpenRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743 in 174 msec 2023-02-11 15:00:31,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:31,347 INFO [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] regionserver.HStore(310): Store=e7f1f013dd1caf6b5254f795ef0a3ed7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:31,347 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN in 333 msec 2023-02-11 15:00:31,347 INFO [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] regionserver.HStore(310): Store=08d6264e7bb30445380a92d3761a3474/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:31,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:31,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:31,348 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN in 335 msec 2023-02-11 15:00:31,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:31,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:31,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:31,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:31,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:31,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:31,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:31,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 43b897ff4788f8ff736d623d593d53f4; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68586078, jitterRate=0.02201220393180847}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:31,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 43b897ff4788f8ff736d623d593d53f4: Running coprocessor pre-open hook at 1676127631340Writing region info on filesystem at 1676127631340Initializing all the Stores at 1676127631342 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127631342Cleaning up temporary data from old regions at 1676127631348 (+6 ms)Cleaning up detritus from prior splits at 1676127631348Running coprocessor post-open hooks at 1676127631353 (+5 ms)Region opened successfully at 1676127631353 2023-02-11 15:00:31,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:31,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4., pid=213, masterSystemTime=1676127631323 2023-02-11 15:00:31,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 15:00:31,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e7f1f013dd1caf6b5254f795ef0a3ed7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72899149, jitterRate=0.08628197014331818}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:31,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e7f1f013dd1caf6b5254f795ef0a3ed7: Running coprocessor pre-open hook at 1676127631342Writing region info on filesystem at 1676127631342Initializing all the Stores at 1676127631342Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127631343 (+1 ms)Cleaning up temporary data from old regions at 1676127631349 (+6 ms)Cleaning up detritus from prior splits at 1676127631349Running coprocessor post-open hooks at 1676127631354 (+5 ms)Region opened successfully at 1676127631354 2023-02-11 15:00:31,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 08d6264e7bb30445380a92d3761a3474; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67551054, jitterRate=0.006589144468307495}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:31,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 08d6264e7bb30445380a92d3761a3474: Running coprocessor pre-open hook at 1676127631342Writing region info on filesystem at 1676127631342Initializing all the Stores at 1676127631343 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127631343Cleaning up temporary data from old regions at 1676127631349 (+6 ms)Cleaning up detritus from prior splits at 1676127631349Running coprocessor post-open hooks at 1676127631354 (+5 ms)Region opened successfully at 1676127631354 2023-02-11 15:00:31,355 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7., pid=214, masterSystemTime=1676127631323 2023-02-11 15:00:31,356 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474., pid=209, masterSystemTime=1676127631322 2023-02-11 15:00:31,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:31,356 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:31,356 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:31,357 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631356"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127631356"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127631356"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127631356"}]},"ts":"1676127631356"} 2023-02-11 15:00:31,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:31,357 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:31,357 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:31,357 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631357"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127631357"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127631357"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127631357"}]},"ts":"1676127631357"} 2023-02-11 15:00:31,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:31,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:31,358 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:31,358 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631358"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127631358"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127631358"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127631358"}]},"ts":"1676127631358"} 2023-02-11 15:00:31,360 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=206 2023-02-11 15:00:31,360 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=206, state=SUCCESS; OpenRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785 in 186 msec 2023-02-11 15:00:31,361 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=204 2023-02-11 15:00:31,361 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=204, state=SUCCESS; OpenRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743 in 184 msec 2023-02-11 15:00:31,364 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=207 2023-02-11 15:00:31,364 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN in 348 msec 2023-02-11 15:00:31,364 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=207, state=SUCCESS; OpenRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675 in 190 msec 2023-02-11 15:00:31,365 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN in 349 msec 2023-02-11 15:00:31,366 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=207, resume processing ppid=202 2023-02-11 15:00:31,366 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN in 352 msec 2023-02-11 15:00:31,366 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676127630926 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 15:00:31,366 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127631366"}]},"ts":"1676127631366"} 2023-02-11 15:00:31,367 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=ENABLED in hbase:meta 2023-02-11 15:00:31,379 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676127630926 execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 15:00:31,381 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676127630926 in 451 msec 2023-02-11 15:00:31,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 15:00:31,684 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676127630926, procId: 202 completed 2023-02-11 15:00:31,686 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676127630926 2023-02-11 15:00:31,686 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:31,689 INFO [Listener at localhost/39605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 15:00:31,701 INFO [Listener at localhost/39605] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676127630926 2023-02-11 15:00:31,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676127630926 2023-02-11 15:00:31,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:31,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-11 15:00:31,705 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127631705"}]},"ts":"1676127631705"} 2023-02-11 15:00:31,706 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=DISABLING in hbase:meta 2023-02-11 15:00:31,709 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676127630926 to state=DISABLING 2023-02-11 15:00:31,709 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, UNASSIGN}] 2023-02-11 15:00:31,710 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, UNASSIGN 2023-02-11 15:00:31,711 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, UNASSIGN 2023-02-11 15:00:31,711 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, UNASSIGN 2023-02-11 15:00:31,711 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, UNASSIGN 2023-02-11 15:00:31,711 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, UNASSIGN 2023-02-11 15:00:31,711 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:31,711 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:31,712 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:31,711 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:31,712 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631711"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631711"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631711"}]},"ts":"1676127631711"} 2023-02-11 15:00:31,712 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127631711"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631711"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631711"}]},"ts":"1676127631711"} 2023-02-11 15:00:31,712 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631711"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631711"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631711"}]},"ts":"1676127631711"} 2023-02-11 15:00:31,712 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631711"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631711"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631711"}]},"ts":"1676127631711"} 2023-02-11 15:00:31,712 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:31,712 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631712"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631712"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631712"}]},"ts":"1676127631712"} 2023-02-11 15:00:31,713 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=220, state=RUNNABLE; CloseRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:31,713 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=221, state=RUNNABLE; CloseRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:31,714 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=218, state=RUNNABLE; CloseRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:31,714 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=219, state=RUNNABLE; CloseRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:31,715 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, UNASSIGN 2023-02-11 15:00:31,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=217, state=RUNNABLE; CloseRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:31,716 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:31,716 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127631716"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127631716"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127631716"}]},"ts":"1676127631716"} 2023-02-11 15:00:31,717 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:31,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:31,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:31,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 08d6264e7bb30445380a92d3761a3474, disabling compactions & flushes 2023-02-11 15:00:31,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8de7ceba3e95ef9d71ef663e0063791d, disabling compactions & flushes 2023-02-11 15:00:31,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:31,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:31,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:31,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:31,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. after waiting 0 ms 2023-02-11 15:00:31,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. after waiting 0 ms 2023-02-11 15:00:31,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:31,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:31,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:31,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a25d52f50b583b8024a28335f497bb9c, disabling compactions & flushes 2023-02-11 15:00:31,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:31,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:31,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. after waiting 0 ms 2023-02-11 15:00:31,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:31,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:31,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:31,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:31,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:31,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 08d6264e7bb30445380a92d3761a3474: Waiting for close lock at 1676127631865Running coprocessor pre-close hooks at 1676127631865Disabling compacts and flushes for region at 1676127631865Disabling writes for close at 1676127631865Writing region close event to WAL at 1676127631865Running coprocessor post-close hooks at 1676127631869 (+4 ms)Closed at 1676127631869 2023-02-11 15:00:31,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:31,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:31,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a25d52f50b583b8024a28335f497bb9c: Waiting for close lock at 1676127631866Running coprocessor pre-close hooks at 1676127631866Disabling compacts and flushes for region at 1676127631866Disabling writes for close at 1676127631866Writing region close event to WAL at 1676127631866Running coprocessor post-close hooks at 1676127631869 (+3 ms)Closed at 1676127631869 2023-02-11 15:00:31,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8de7ceba3e95ef9d71ef663e0063791d: Waiting for close lock at 1676127631865Running coprocessor pre-close hooks at 1676127631865Disabling compacts and flushes for region at 1676127631865Disabling writes for close at 1676127631865Writing region close event to WAL at 1676127631865Running coprocessor post-close hooks at 1676127631869 (+4 ms)Closed at 1676127631869 2023-02-11 15:00:31,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:31,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:31,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 238478a8a495efd2d422ce3a8631bbc2, disabling compactions & flushes 2023-02-11 15:00:31,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:31,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:31,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. after waiting 0 ms 2023-02-11 15:00:31,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:31,871 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=CLOSED 2023-02-11 15:00:31,871 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631871"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127631871"}]},"ts":"1676127631871"} 2023-02-11 15:00:31,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:31,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:31,871 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 43b897ff4788f8ff736d623d593d53f4, disabling compactions & flushes 2023-02-11 15:00:31,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:31,871 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:31,871 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. after waiting 0 ms 2023-02-11 15:00:31,871 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:31,871 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=CLOSED 2023-02-11 15:00:31,872 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631871"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127631871"}]},"ts":"1676127631871"} 2023-02-11 15:00:31,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:31,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:31,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e7f1f013dd1caf6b5254f795ef0a3ed7, disabling compactions & flushes 2023-02-11 15:00:31,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:31,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:31,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. after waiting 0 ms 2023-02-11 15:00:31,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:31,873 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=CLOSED 2023-02-11 15:00:31,873 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127631873"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127631873"}]},"ts":"1676127631873"} 2023-02-11 15:00:31,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:31,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:31,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:31,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 238478a8a495efd2d422ce3a8631bbc2: Waiting for close lock at 1676127631870Running coprocessor pre-close hooks at 1676127631870Disabling compacts and flushes for region at 1676127631870Disabling writes for close at 1676127631870Writing region close event to WAL at 1676127631871 (+1 ms)Running coprocessor post-close hooks at 1676127631875 (+4 ms)Closed at 1676127631875 2023-02-11 15:00:31,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 15:00:31,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:31,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 43b897ff4788f8ff736d623d593d53f4: Waiting for close lock at 1676127631871Running coprocessor pre-close hooks at 1676127631871Disabling compacts and flushes for region at 1676127631871Disabling writes for close at 1676127631871Writing region close event to WAL at 1676127631872 (+1 ms)Running coprocessor post-close hooks at 1676127631875 (+3 ms)Closed at 1676127631875 2023-02-11 15:00:31,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:31,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e7f1f013dd1caf6b5254f795ef0a3ed7: Waiting for close lock at 1676127631872Running coprocessor pre-close hooks at 1676127631872Disabling compacts and flushes for region at 1676127631872Disabling writes for close at 1676127631872Writing region close event to WAL at 1676127631872Running coprocessor post-close hooks at 1676127631875 (+3 ms)Closed at 1676127631875 2023-02-11 15:00:31,877 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=220 2023-02-11 15:00:31,877 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=220, state=SUCCESS; CloseRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675 in 159 msec 2023-02-11 15:00:31,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:31,879 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=218 2023-02-11 15:00:31,879 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=218, state=SUCCESS; CloseRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785 in 159 msec 2023-02-11 15:00:31,880 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=CLOSED 2023-02-11 15:00:31,880 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631880"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127631880"}]},"ts":"1676127631880"} 2023-02-11 15:00:31,880 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:31,880 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=221 2023-02-11 15:00:31,880 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, UNASSIGN in 168 msec 2023-02-11 15:00:31,880 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=221, state=SUCCESS; CloseRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743 in 164 msec 2023-02-11 15:00:31,880 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=CLOSED 2023-02-11 15:00:31,881 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127631880"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127631880"}]},"ts":"1676127631880"} 2023-02-11 15:00:31,881 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:31,881 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, UNASSIGN in 170 msec 2023-02-11 15:00:31,881 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=CLOSED 2023-02-11 15:00:31,882 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127631881"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127631881"}]},"ts":"1676127631881"} 2023-02-11 15:00:31,882 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, UNASSIGN in 171 msec 2023-02-11 15:00:31,884 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=217 2023-02-11 15:00:31,884 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=217, state=SUCCESS; CloseRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743 in 167 msec 2023-02-11 15:00:31,885 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=219 2023-02-11 15:00:31,885 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=219, state=SUCCESS; CloseRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785 in 169 msec 2023-02-11 15:00:31,885 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-02-11 15:00:31,885 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, UNASSIGN in 175 msec 2023-02-11 15:00:31,885 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675 in 166 msec 2023-02-11 15:00:31,885 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, UNASSIGN in 176 msec 2023-02-11 15:00:31,886 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=216, resume processing ppid=215 2023-02-11 15:00:31,886 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, UNASSIGN in 176 msec 2023-02-11 15:00:31,887 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127631887"}]},"ts":"1676127631887"} 2023-02-11 15:00:31,888 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=DISABLED in hbase:meta 2023-02-11 15:00:31,889 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676127630926 to state=DISABLED 2023-02-11 15:00:31,892 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676127630926 in 190 msec 2023-02-11 15:00:31,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-11 15:00:31,956 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676127630926, procId: 215 completed 2023-02-11 15:00:31,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 } 2023-02-11 15:00:31,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676127631956 (current time:1676127631956). 2023-02-11 15:00:31,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 15:00:31,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(139): Connect 0x18485525 to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:31,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6856cfa2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:31,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:31,964 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50694, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:31,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18485525 to 127.0.0.1:58940 2023-02-11 15:00:31,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:31,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 15:00:31,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 } 2023-02-11 15:00:31,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676127630926, type=EXCLUSIVE 2023-02-11 15:00:31,968 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676127630926, type=EXCLUSIVE 2023-02-11 15:00:31,969 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676127630926, type=EXCLUSIVE; timeout=600000, timestamp=1676128231969 2023-02-11 15:00:31,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 } 2023-02-11 15:00:31,969 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676127630926 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676127630926 2023-02-11 15:00:31,969 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 15:00:31,969 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 15:00:31,978 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676127630926 table=testCorruptedSnapshot-1676127630926 type=DISABLED ttl=0 } 2023-02-11 15:00:31,984 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:31,984 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:31,984 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:31,984 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:31,984 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:31,984 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:31,985 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:31,985 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:31,985 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:31,985 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:31,985 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:31,985 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:31,985 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:31,985 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:31,989 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:31,989 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:31,989 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:31,990 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:31,990 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:31,990 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:31,992 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 15:00:32,008 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 15:00:32,056 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676127630926 table=testCorruptedSnapshot-1676127630926 type=DISABLED ttl=0 } as finished. 2023-02-11 15:00:32,056 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676127630926 2023-02-11 15:00:32,056 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/emptySnaptb-1676127630926 2023-02-11 15:00:32,079 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/emptySnaptb-1676127630926 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/emptySnaptb-1676127630926 2023-02-11 15:00:32,095 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676127630926 of table testCorruptedSnapshot-1676127630926 completed 2023-02-11 15:00:32,096 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/emptySnaptb-1676127630926 2023-02-11 15:00:32,096 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/emptySnaptb-1676127630926 2023-02-11 15:00:32,096 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676127630926 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676127630926 at 1676127631969Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676127630926 table=testCorruptedSnapshot-1676127630926 type=DISABLED ttl=0 } at 1676127631979 (+10 ms)Consolidate snapshot: emptySnaptb-1676127630926 at 1676127632056 (+77 ms)Loading Region manifests for emptySnaptb-1676127630926 at 1676127632056Writing data manifest for emptySnaptb-1676127630926 at 1676127632064 (+8 ms)Verifying snapshot: emptySnaptb-1676127630926 at 1676127632075 (+11 ms)Snapshot emptySnaptb-1676127630926 of table testCorruptedSnapshot-1676127630926 completed at 1676127632095 (+20 ms) 2023-02-11 15:00:32,099 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676127630926, type=EXCLUSIVE 2023-02-11 15:00:32,100 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676127630926, type=EXCLUSIVE in 132 msec 2023-02-11 15:00:32,220 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 15:00:32,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 } is done 2023-02-11 15:00:32,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 15:00:32,221 INFO [Listener at localhost/39605] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676127630926 2023-02-11 15:00:32,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676127630926 2023-02-11 15:00:32,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:32,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 15:00:32,224 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676127630926 2023-02-11 15:00:32,225 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127632224"}]},"ts":"1676127632224"} 2023-02-11 15:00:32,226 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=ENABLING in hbase:meta 2023-02-11 15:00:32,229 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 15:00:32,229 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN}] 2023-02-11 15:00:32,230 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN 2023-02-11 15:00:32,230 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN 2023-02-11 15:00:32,230 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN 2023-02-11 15:00:32,230 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN 2023-02-11 15:00:32,230 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:32,230 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=true 2023-02-11 15:00:32,231 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:32,231 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=true 2023-02-11 15:00:32,231 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:32,231 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN 2023-02-11 15:00:32,231 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=true 2023-02-11 15:00:32,231 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN 2023-02-11 15:00:32,231 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:32,231 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=true 2023-02-11 15:00:32,232 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:32,232 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:32,232 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=true 2023-02-11 15:00:32,232 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=true 2023-02-11 15:00:32,381 INFO [jenkins-hbase4:38517] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 15:00:32,381 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:32,381 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:32,381 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:32,381 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:32,381 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:32,381 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127632381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127632381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127632381"}]},"ts":"1676127632381"} 2023-02-11 15:00:32,381 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127632381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127632381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127632381"}]},"ts":"1676127632381"} 2023-02-11 15:00:32,381 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127632381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127632381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127632381"}]},"ts":"1676127632381"} 2023-02-11 15:00:32,381 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127632381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127632381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127632381"}]},"ts":"1676127632381"} 2023-02-11 15:00:32,381 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127632381"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127632381"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127632381"}]},"ts":"1676127632381"} 2023-02-11 15:00:32,383 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=230, state=RUNNABLE; OpenRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:32,383 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=233, state=RUNNABLE; OpenRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:32,384 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=235, state=RUNNABLE; OpenRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:32,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=234, state=RUNNABLE; OpenRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:32,385 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=232, state=RUNNABLE; OpenRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:32,385 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:32,385 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127632385"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127632385"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127632385"}]},"ts":"1676127632385"} 2023-02-11 15:00:32,386 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=231, state=RUNNABLE; OpenRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:32,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 15:00:32,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:32,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 08d6264e7bb30445380a92d3761a3474, NAME => 'testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 15:00:32,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:32,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:32,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:32,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:32,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:32,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8de7ceba3e95ef9d71ef663e0063791d, NAME => 'testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 15:00:32,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:32,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:32,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:32,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:32,538 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:32,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 43b897ff4788f8ff736d623d593d53f4, NAME => 'testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 15:00:32,538 INFO [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:32,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:32,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:32,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:32,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:32,539 INFO [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:32,539 DEBUG [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf 2023-02-11 15:00:32,539 DEBUG [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf 2023-02-11 15:00:32,539 INFO [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08d6264e7bb30445380a92d3761a3474 columnFamilyName cf 2023-02-11 15:00:32,539 INFO [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:32,539 DEBUG [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf 2023-02-11 15:00:32,540 DEBUG [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf 2023-02-11 15:00:32,540 INFO [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8de7ceba3e95ef9d71ef663e0063791d columnFamilyName cf 2023-02-11 15:00:32,540 INFO [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] regionserver.HStore(310): Store=08d6264e7bb30445380a92d3761a3474/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:32,540 DEBUG [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf 2023-02-11 15:00:32,540 DEBUG [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf 2023-02-11 15:00:32,541 INFO [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 43b897ff4788f8ff736d623d593d53f4 columnFamilyName cf 2023-02-11 15:00:32,541 INFO [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] regionserver.HStore(310): Store=8de7ceba3e95ef9d71ef663e0063791d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:32,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:32,541 INFO [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] regionserver.HStore(310): Store=43b897ff4788f8ff736d623d593d53f4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:32,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:32,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:32,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:32,543 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:32,543 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:32,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:32,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 08d6264e7bb30445380a92d3761a3474; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64023982, jitterRate=-0.045968323945999146}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:32,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 08d6264e7bb30445380a92d3761a3474: Running coprocessor pre-open hook at 1676127632537Writing region info on filesystem at 1676127632537Initializing all the Stores at 1676127632538 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127632538Cleaning up temporary data from old regions at 1676127632542 (+4 ms)Cleaning up detritus from prior splits at 1676127632543 (+1 ms)Running coprocessor post-open hooks at 1676127632545 (+2 ms)Region opened successfully at 1676127632545 2023-02-11 15:00:32,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:32,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:32,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474., pid=239, masterSystemTime=1676127632534 2023-02-11 15:00:32,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8de7ceba3e95ef9d71ef663e0063791d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61818545, jitterRate=-0.07883189618587494}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:32,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 43b897ff4788f8ff736d623d593d53f4; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73715392, jitterRate=0.09844493865966797}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:32,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8de7ceba3e95ef9d71ef663e0063791d: Running coprocessor pre-open hook at 1676127632538Writing region info on filesystem at 1676127632538Initializing all the Stores at 1676127632538Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127632538Cleaning up temporary data from old regions at 1676127632543 (+5 ms)Cleaning up detritus from prior splits at 1676127632544 (+1 ms)Running coprocessor post-open hooks at 1676127632546 (+2 ms)Region opened successfully at 1676127632546 2023-02-11 15:00:32,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 43b897ff4788f8ff736d623d593d53f4: Running coprocessor pre-open hook at 1676127632539Writing region info on filesystem at 1676127632539Initializing all the Stores at 1676127632539Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127632539Cleaning up temporary data from old regions at 1676127632544 (+5 ms)Cleaning up detritus from prior splits at 1676127632544Running coprocessor post-open hooks at 1676127632546 (+2 ms)Region opened successfully at 1676127632546 2023-02-11 15:00:32,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d., pid=238, masterSystemTime=1676127632535 2023-02-11 15:00:32,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4., pid=237, masterSystemTime=1676127632535 2023-02-11 15:00:32,547 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:32,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:32,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:32,547 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 238478a8a495efd2d422ce3a8631bbc2, NAME => 'testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 15:00:32,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:32,548 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:32,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:32,548 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127632548"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127632548"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127632548"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127632548"}]},"ts":"1676127632548"} 2023-02-11 15:00:32,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:32,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:32,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:32,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:32,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:32,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a25d52f50b583b8024a28335f497bb9c, NAME => 'testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 15:00:32,549 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:32,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:32,549 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127632548"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127632548"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127632548"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127632548"}]},"ts":"1676127632548"} 2023-02-11 15:00:32,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:32,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:32,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:32,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:32,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:32,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:32,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e7f1f013dd1caf6b5254f795ef0a3ed7, NAME => 'testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 15:00:32,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:32,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:32,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:32,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:32,550 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:32,550 INFO [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:32,550 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127632550"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127632550"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127632550"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127632550"}]},"ts":"1676127632550"} 2023-02-11 15:00:32,551 DEBUG [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf 2023-02-11 15:00:32,551 DEBUG [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf 2023-02-11 15:00:32,551 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=234 2023-02-11 15:00:32,551 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=234, state=SUCCESS; OpenRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675 in 165 msec 2023-02-11 15:00:32,552 INFO [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:32,552 INFO [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:32,552 INFO [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 238478a8a495efd2d422ce3a8631bbc2 columnFamilyName cf 2023-02-11 15:00:32,552 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=233 2023-02-11 15:00:32,552 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=233, state=SUCCESS; OpenRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785 in 167 msec 2023-02-11 15:00:32,553 INFO [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] regionserver.HStore(310): Store=238478a8a495efd2d422ce3a8631bbc2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:32,553 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN in 322 msec 2023-02-11 15:00:32,553 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=235 2023-02-11 15:00:32,553 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=235, state=SUCCESS; OpenRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743 in 167 msec 2023-02-11 15:00:32,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:32,554 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN in 324 msec 2023-02-11 15:00:32,554 DEBUG [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf 2023-02-11 15:00:32,554 DEBUG [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf 2023-02-11 15:00:32,554 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN in 324 msec 2023-02-11 15:00:32,554 DEBUG [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf 2023-02-11 15:00:32,554 DEBUG [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf 2023-02-11 15:00:32,555 INFO [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7f1f013dd1caf6b5254f795ef0a3ed7 columnFamilyName cf 2023-02-11 15:00:32,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:32,555 INFO [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] regionserver.HStore(310): Store=e7f1f013dd1caf6b5254f795ef0a3ed7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:32,556 INFO [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a25d52f50b583b8024a28335f497bb9c columnFamilyName cf 2023-02-11 15:00:32,556 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:32,557 INFO [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] regionserver.HStore(310): Store=a25d52f50b583b8024a28335f497bb9c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:32,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:32,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:32,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:32,558 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 238478a8a495efd2d422ce3a8631bbc2; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73862965, jitterRate=0.10064394772052765}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:32,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 238478a8a495efd2d422ce3a8631bbc2: Running coprocessor pre-open hook at 1676127632548Writing region info on filesystem at 1676127632548Initializing all the Stores at 1676127632550 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127632550Cleaning up temporary data from old regions at 1676127632555 (+5 ms)Cleaning up detritus from prior splits at 1676127632555Running coprocessor post-open hooks at 1676127632558 (+3 ms)Region opened successfully at 1676127632558 2023-02-11 15:00:32,558 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2., pid=236, masterSystemTime=1676127632534 2023-02-11 15:00:32,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:32,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:32,560 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e7f1f013dd1caf6b5254f795ef0a3ed7; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60048917, jitterRate=-0.10520140826702118}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:32,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e7f1f013dd1caf6b5254f795ef0a3ed7: Running coprocessor pre-open hook at 1676127632550Writing region info on filesystem at 1676127632550Initializing all the Stores at 1676127632550Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127632550Cleaning up temporary data from old regions at 1676127632558 (+8 ms)Cleaning up detritus from prior splits at 1676127632558Running coprocessor post-open hooks at 1676127632560 (+2 ms)Region opened successfully at 1676127632560 2023-02-11 15:00:32,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:32,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:32,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7., pid=241, masterSystemTime=1676127632535 2023-02-11 15:00:32,561 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:32,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:32,561 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127632561"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127632561"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127632561"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127632561"}]},"ts":"1676127632561"} 2023-02-11 15:00:32,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:32,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a25d52f50b583b8024a28335f497bb9c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61711554, jitterRate=-0.0804261863231659}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:32,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:32,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a25d52f50b583b8024a28335f497bb9c: Running coprocessor pre-open hook at 1676127632549Writing region info on filesystem at 1676127632549Initializing all the Stores at 1676127632550 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127632551 (+1 ms)Cleaning up temporary data from old regions at 1676127632559 (+8 ms)Cleaning up detritus from prior splits at 1676127632559Running coprocessor post-open hooks at 1676127632562 (+3 ms)Region opened successfully at 1676127632562 2023-02-11 15:00:32,563 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:32,563 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127632563"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127632563"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127632563"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127632563"}]},"ts":"1676127632563"} 2023-02-11 15:00:32,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c., pid=240, masterSystemTime=1676127632535 2023-02-11 15:00:32,564 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=230 2023-02-11 15:00:32,564 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=230, state=SUCCESS; OpenRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675 in 179 msec 2023-02-11 15:00:32,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:32,565 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:32,565 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:32,565 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127632565"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127632565"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127632565"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127632565"}]},"ts":"1676127632565"} 2023-02-11 15:00:32,566 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN in 335 msec 2023-02-11 15:00:32,566 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=231 2023-02-11 15:00:32,566 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=231, state=SUCCESS; OpenRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743 in 178 msec 2023-02-11 15:00:32,567 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN in 337 msec 2023-02-11 15:00:32,568 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=232 2023-02-11 15:00:32,568 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=232, state=SUCCESS; OpenRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785 in 182 msec 2023-02-11 15:00:32,569 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=232, resume processing ppid=229 2023-02-11 15:00:32,569 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN in 339 msec 2023-02-11 15:00:32,569 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127632569"}]},"ts":"1676127632569"} 2023-02-11 15:00:32,570 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=ENABLED in hbase:meta 2023-02-11 15:00:32,572 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676127630926' was successfully enabled. 2023-02-11 15:00:32,573 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676127630926 in 351 msec 2023-02-11 15:00:32,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 15:00:32,976 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676127630926, procId: 229 completed 2023-02-11 15:00:32,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:32,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:32,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:32,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:32,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:32,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:32,991 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676127630926 2023-02-11 15:00:32,991 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:32,991 INFO [Listener at localhost/39605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 15:00:33,003 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 15:00:33,004 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 15:00:33,005 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 15:00:33,006 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 15:00:33,007 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 15:00:33,009 INFO [Listener at localhost/39605] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676127630926 2023-02-11 15:00:33,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676127630926 2023-02-11 15:00:33,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:33,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 15:00:33,013 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127633013"}]},"ts":"1676127633013"} 2023-02-11 15:00:33,014 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=DISABLING in hbase:meta 2023-02-11 15:00:33,016 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676127630926 to state=DISABLING 2023-02-11 15:00:33,016 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, UNASSIGN}] 2023-02-11 15:00:33,017 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, UNASSIGN 2023-02-11 15:00:33,017 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, UNASSIGN 2023-02-11 15:00:33,017 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, UNASSIGN 2023-02-11 15:00:33,017 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, UNASSIGN 2023-02-11 15:00:33,018 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, UNASSIGN 2023-02-11 15:00:33,018 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:33,018 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:33,018 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127633018"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127633018"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127633018"}]},"ts":"1676127633018"} 2023-02-11 15:00:33,018 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127633018"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127633018"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127633018"}]},"ts":"1676127633018"} 2023-02-11 15:00:33,018 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:33,018 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:33,018 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127633018"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127633018"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127633018"}]},"ts":"1676127633018"} 2023-02-11 15:00:33,018 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127633018"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127633018"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127633018"}]},"ts":"1676127633018"} 2023-02-11 15:00:33,018 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:33,019 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127633018"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127633018"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127633018"}]},"ts":"1676127633018"} 2023-02-11 15:00:33,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=245, state=RUNNABLE; CloseRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:33,020 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=247, state=RUNNABLE; CloseRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:33,020 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=248, state=RUNNABLE; CloseRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:33,021 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=246, state=RUNNABLE; CloseRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:33,022 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=244, state=RUNNABLE; CloseRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:33,023 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, UNASSIGN 2023-02-11 15:00:33,024 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:33,024 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127633024"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127633024"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127633024"}]},"ts":"1676127633024"} 2023-02-11 15:00:33,025 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=243, state=RUNNABLE; CloseRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:33,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:33,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:33,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a25d52f50b583b8024a28335f497bb9c, disabling compactions & flushes 2023-02-11 15:00:33,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 08d6264e7bb30445380a92d3761a3474, disabling compactions & flushes 2023-02-11 15:00:33,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:33,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:33,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:33,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:33,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. after waiting 0 ms 2023-02-11 15:00:33,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:33,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. after waiting 0 ms 2023-02-11 15:00:33,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:33,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 08d6264e7bb30445380a92d3761a3474 1/1 column families, dataSize=2.48 KB heapSize=5.59 KB 2023-02-11 15:00:33,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a25d52f50b583b8024a28335f497bb9c 1/1 column families, dataSize=1.63 KB heapSize=3.77 KB 2023-02-11 15:00:33,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:33,173 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e7f1f013dd1caf6b5254f795ef0a3ed7, disabling compactions & flushes 2023-02-11 15:00:33,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:33,173 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:33,173 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. after waiting 0 ms 2023-02-11 15:00:33,173 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:33,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e7f1f013dd1caf6b5254f795ef0a3ed7 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-11 15:00:33,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230211c776e56342844884b1e403356e08725c to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c4ca4238a0b923820dcc509a6f75849b20230211c776e56342844884b1e403356e08725c 2023-02-11 15:00:33,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302110a80c9c9d4ca41218546e28ae6398848 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/a87ff679a2f3e71d9181a67b7542122c202302110a80c9c9d4ca41218546e28ae6398848 2023-02-11 15:00:33,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c2023021183f238ac4534473e881d6ebe70c1e423 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021183f238ac4534473e881d6ebe70c1e423 2023-02-11 15:00:33,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/.tmp/cf/2d2aeada4c3d44b2a6b3e855d8a6a262 2023-02-11 15:00:33,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/.tmp/cf/708f88c8016544edb7fe2ac974bc6a97 2023-02-11 15:00:33,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/.tmp/cf/2d2aeada4c3d44b2a6b3e855d8a6a262 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2d2aeada4c3d44b2a6b3e855d8a6a262 2023-02-11 15:00:33,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/.tmp/cf/708f88c8016544edb7fe2ac974bc6a97 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/708f88c8016544edb7fe2ac974bc6a97 2023-02-11 15:00:33,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/.tmp/cf/7ae84fa17bc340af8afaef45c904e668 2023-02-11 15:00:33,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2d2aeada4c3d44b2a6b3e855d8a6a262, entries=38, sequenceid=8, filesize=11.4 K 2023-02-11 15:00:33,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for 08d6264e7bb30445380a92d3761a3474 in 38ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:33,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676127630926' 2023-02-11 15:00:33,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/708f88c8016544edb7fe2ac974bc6a97, entries=27, sequenceid=8, filesize=9.6 K 2023-02-11 15:00:33,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for e7f1f013dd1caf6b5254f795ef0a3ed7 in 39ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:33,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/.tmp/cf/7ae84fa17bc340af8afaef45c904e668 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/7ae84fa17bc340af8afaef45c904e668 2023-02-11 15:00:33,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:33,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:33,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:33,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 08d6264e7bb30445380a92d3761a3474: Waiting for close lock at 1676127633172Running coprocessor pre-close hooks at 1676127633172Disabling compacts and flushes for region at 1676127633172Disabling writes for close at 1676127633172Obtaining lock to block concurrent updates at 1676127633173 (+1 ms)Preparing flush snapshotting stores in 08d6264e7bb30445380a92d3761a3474 at 1676127633173Finished memstore snapshotting testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474., syncing WAL and waiting on mvcc, flushsize=dataSize=2544, getHeapSize=5712, getOffHeapSize=0, getCellsCount=38 at 1676127633173Flushing stores of testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. at 1676127633173Flushing 08d6264e7bb30445380a92d3761a3474/cf: creating writer at 1676127633174 (+1 ms)Flushing mob file 08d6264e7bb30445380a92d3761a3474/cf: appending metadata at 1676127633179 (+5 ms)Flushing mob file 08d6264e7bb30445380a92d3761a3474/cf: closing flushed file at 1676127633179Flushing 08d6264e7bb30445380a92d3761a3474/cf: appending metadata at 1676127633193 (+14 ms)Flushing 08d6264e7bb30445380a92d3761a3474/cf: closing flushed file at 1676127633194 (+1 ms)Flushing 08d6264e7bb30445380a92d3761a3474/cf: reopening flushed file at 1676127633206 (+12 ms)Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for 08d6264e7bb30445380a92d3761a3474 in 38ms, sequenceid=8, compaction requested=false at 1676127633210 (+4 ms)Writing region close event to WAL at 1676127633211 (+1 ms)Running coprocessor post-close hooks at 1676127633216 (+5 ms)Closed at 1676127633216 2023-02-11 15:00:33,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:33,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e7f1f013dd1caf6b5254f795ef0a3ed7: Waiting for close lock at 1676127633173Running coprocessor pre-close hooks at 1676127633173Disabling compacts and flushes for region at 1676127633173Disabling writes for close at 1676127633173Obtaining lock to block concurrent updates at 1676127633174 (+1 ms)Preparing flush snapshotting stores in e7f1f013dd1caf6b5254f795ef0a3ed7 at 1676127633174Finished memstore snapshotting testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676127633174Flushing stores of testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. at 1676127633175 (+1 ms)Flushing e7f1f013dd1caf6b5254f795ef0a3ed7/cf: creating writer at 1676127633176 (+1 ms)Flushing mob file e7f1f013dd1caf6b5254f795ef0a3ed7/cf: appending metadata at 1676127633179 (+3 ms)Flushing mob file e7f1f013dd1caf6b5254f795ef0a3ed7/cf: closing flushed file at 1676127633179Flushing e7f1f013dd1caf6b5254f795ef0a3ed7/cf: appending metadata at 1676127633193 (+14 ms)Flushing e7f1f013dd1caf6b5254f795ef0a3ed7/cf: closing flushed file at 1676127633194 (+1 ms)Flushing e7f1f013dd1caf6b5254f795ef0a3ed7/cf: reopening flushed file at 1676127633207 (+13 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for e7f1f013dd1caf6b5254f795ef0a3ed7 in 39ms, sequenceid=8, compaction requested=false at 1676127633212 (+5 ms)Writing region close event to WAL at 1676127633213 (+1 ms)Running coprocessor post-close hooks at 1676127633217 (+4 ms)Closed at 1676127633217 2023-02-11 15:00:33,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:33,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:33,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 238478a8a495efd2d422ce3a8631bbc2, disabling compactions & flushes 2023-02-11 15:00:33,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:33,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:33,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. after waiting 0 ms 2023-02-11 15:00:33,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:33,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/7ae84fa17bc340af8afaef45c904e668, entries=25, sequenceid=8, filesize=9.2 K 2023-02-11 15:00:33,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 238478a8a495efd2d422ce3a8631bbc2 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-11 15:00:33,218 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=CLOSED 2023-02-11 15:00:33,219 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127633218"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127633218"}]},"ts":"1676127633218"} 2023-02-11 15:00:33,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for a25d52f50b583b8024a28335f497bb9c in 47ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:33,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:33,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:33,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8de7ceba3e95ef9d71ef663e0063791d, disabling compactions & flushes 2023-02-11 15:00:33,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:33,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:33,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. after waiting 0 ms 2023-02-11 15:00:33,221 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=CLOSED 2023-02-11 15:00:33,221 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127633221"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127633221"}]},"ts":"1676127633221"} 2023-02-11 15:00:33,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:33,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 8de7ceba3e95ef9d71ef663e0063791d 1/1 column families, dataSize=22.18 KB heapSize=47.92 KB 2023-02-11 15:00:33,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=247 2023-02-11 15:00:33,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=247, state=SUCCESS; CloseRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675 in 201 msec 2023-02-11 15:00:33,230 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=244 2023-02-11 15:00:33,230 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:33,230 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=244, state=SUCCESS; CloseRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743 in 200 msec 2023-02-11 15:00:33,230 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, UNASSIGN in 212 msec 2023-02-11 15:00:33,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:33,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a25d52f50b583b8024a28335f497bb9c: Waiting for close lock at 1676127633172Running coprocessor pre-close hooks at 1676127633172Disabling compacts and flushes for region at 1676127633172Disabling writes for close at 1676127633172Obtaining lock to block concurrent updates at 1676127633173 (+1 ms)Preparing flush snapshotting stores in a25d52f50b583b8024a28335f497bb9c at 1676127633173Finished memstore snapshotting testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c., syncing WAL and waiting on mvcc, flushsize=dataSize=1673, getHeapSize=3840, getOffHeapSize=0, getCellsCount=25 at 1676127633173Flushing stores of testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. at 1676127633173Flushing a25d52f50b583b8024a28335f497bb9c/cf: creating writer at 1676127633174 (+1 ms)Flushing mob file a25d52f50b583b8024a28335f497bb9c/cf: appending metadata at 1676127633179 (+5 ms)Flushing mob file a25d52f50b583b8024a28335f497bb9c/cf: closing flushed file at 1676127633179Flushing a25d52f50b583b8024a28335f497bb9c/cf: appending metadata at 1676127633194 (+15 ms)Flushing a25d52f50b583b8024a28335f497bb9c/cf: closing flushed file at 1676127633194Flushing a25d52f50b583b8024a28335f497bb9c/cf: reopening flushed file at 1676127633216 (+22 ms)Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for a25d52f50b583b8024a28335f497bb9c in 47ms, sequenceid=8, compaction requested=false at 1676127633219 (+3 ms)Writing region close event to WAL at 1676127633222 (+3 ms)Running coprocessor post-close hooks at 1676127633231 (+9 ms)Closed at 1676127633231 2023-02-11 15:00:33,232 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, UNASSIGN in 214 msec 2023-02-11 15:00:33,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:33,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:33,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 43b897ff4788f8ff736d623d593d53f4, disabling compactions & flushes 2023-02-11 15:00:33,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:33,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:33,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. after waiting 0 ms 2023-02-11 15:00:33,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:33,232 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=CLOSED 2023-02-11 15:00:33,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 43b897ff4788f8ff736d623d593d53f4 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-11 15:00:33,233 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127633232"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127633232"}]},"ts":"1676127633232"} 2023-02-11 15:00:33,235 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=245 2023-02-11 15:00:33,235 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=245, state=SUCCESS; CloseRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785 in 215 msec 2023-02-11 15:00:33,235 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, UNASSIGN in 219 msec 2023-02-11 15:00:33,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302118aa224b08bea443c9064e5c096ba8141 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/d41d8cd98f00b204e9800998ecf8427e202302118aa224b08bea443c9064e5c096ba8141 2023-02-11 15:00:33,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230211049aa6a59ea446b7bd306f22bffc8da1 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211049aa6a59ea446b7bd306f22bffc8da1 2023-02-11 15:00:33,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230211c1ae9b317a04417999a6bf1fa35fe905 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/e4da3b7fbbce2345d7772b0674a318d520230211c1ae9b317a04417999a6bf1fa35fe905 2023-02-11 15:00:33,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 15:00:33,268 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/.tmp/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf 2023-02-11 15:00:33,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/.tmp/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf 2023-02-11 15:00:33,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf, entries=32, sequenceid=8, filesize=10.4 K 2023-02-11 15:00:33,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/.tmp/cf/4e63262457624e1c89e80e00eb7444c1 2023-02-11 15:00:33,275 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 238478a8a495efd2d422ce3a8631bbc2 in 57ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:33,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/.tmp/cf/098b7a453bed4669b163aa8c746c8c95 2023-02-11 15:00:33,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/.tmp/cf/4e63262457624e1c89e80e00eb7444c1 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/4e63262457624e1c89e80e00eb7444c1 2023-02-11 15:00:33,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:33,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:33,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 238478a8a495efd2d422ce3a8631bbc2: Waiting for close lock at 1676127633218Running coprocessor pre-close hooks at 1676127633218Disabling compacts and flushes for region at 1676127633218Disabling writes for close at 1676127633218Obtaining lock to block concurrent updates at 1676127633218Preparing flush snapshotting stores in 238478a8a495efd2d422ce3a8631bbc2 at 1676127633218Finished memstore snapshotting testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676127633219 (+1 ms)Flushing stores of testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. at 1676127633219Flushing 238478a8a495efd2d422ce3a8631bbc2/cf: creating writer at 1676127633221 (+2 ms)Flushing mob file 238478a8a495efd2d422ce3a8631bbc2/cf: appending metadata at 1676127633230 (+9 ms)Flushing mob file 238478a8a495efd2d422ce3a8631bbc2/cf: closing flushed file at 1676127633230Flushing 238478a8a495efd2d422ce3a8631bbc2/cf: appending metadata at 1676127633258 (+28 ms)Flushing 238478a8a495efd2d422ce3a8631bbc2/cf: closing flushed file at 1676127633258Flushing 238478a8a495efd2d422ce3a8631bbc2/cf: reopening flushed file at 1676127633271 (+13 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 238478a8a495efd2d422ce3a8631bbc2 in 57ms, sequenceid=8, compaction requested=false at 1676127633275 (+4 ms)Writing region close event to WAL at 1676127633276 (+1 ms)Running coprocessor post-close hooks at 1676127633279 (+3 ms)Closed at 1676127633279 2023-02-11 15:00:33,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/.tmp/cf/098b7a453bed4669b163aa8c746c8c95 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/098b7a453bed4669b163aa8c746c8c95 2023-02-11 15:00:33,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:33,281 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=CLOSED 2023-02-11 15:00:33,281 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127633281"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127633281"}]},"ts":"1676127633281"} 2023-02-11 15:00:33,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/4e63262457624e1c89e80e00eb7444c1, entries=39, sequenceid=8, filesize=11.6 K 2023-02-11 15:00:33,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 43b897ff4788f8ff736d623d593d53f4 in 50ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:33,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/098b7a453bed4669b163aa8c746c8c95, entries=339, sequenceid=8, filesize=63.3 K 2023-02-11 15:00:33,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.18 KB/22709, heapSize ~47.91 KB/49056, currentSize=0 B/0 for 8de7ceba3e95ef9d71ef663e0063791d in 63ms, sequenceid=8, compaction requested=false 2023-02-11 15:00:33,284 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=243 2023-02-11 15:00:33,284 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=243, state=SUCCESS; CloseRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675 in 257 msec 2023-02-11 15:00:33,285 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, UNASSIGN in 268 msec 2023-02-11 15:00:33,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:33,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:33,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 43b897ff4788f8ff736d623d593d53f4: Waiting for close lock at 1676127633232Running coprocessor pre-close hooks at 1676127633232Disabling compacts and flushes for region at 1676127633232Disabling writes for close at 1676127633232Obtaining lock to block concurrent updates at 1676127633233 (+1 ms)Preparing flush snapshotting stores in 43b897ff4788f8ff736d623d593d53f4 at 1676127633233Finished memstore snapshotting testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676127633233Flushing stores of testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. at 1676127633233Flushing 43b897ff4788f8ff736d623d593d53f4/cf: creating writer at 1676127633238 (+5 ms)Flushing mob file 43b897ff4788f8ff736d623d593d53f4/cf: appending metadata at 1676127633248 (+10 ms)Flushing mob file 43b897ff4788f8ff736d623d593d53f4/cf: closing flushed file at 1676127633248Flushing 43b897ff4788f8ff736d623d593d53f4/cf: appending metadata at 1676127633260 (+12 ms)Flushing 43b897ff4788f8ff736d623d593d53f4/cf: closing flushed file at 1676127633260Flushing 43b897ff4788f8ff736d623d593d53f4/cf: reopening flushed file at 1676127633279 (+19 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 43b897ff4788f8ff736d623d593d53f4 in 50ms, sequenceid=8, compaction requested=false at 1676127633282 (+3 ms)Writing region close event to WAL at 1676127633283 (+1 ms)Running coprocessor post-close hooks at 1676127633287 (+4 ms)Closed at 1676127633287 2023-02-11 15:00:33,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:33,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 15:00:33,288 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=CLOSED 2023-02-11 15:00:33,288 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127633288"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127633288"}]},"ts":"1676127633288"} 2023-02-11 15:00:33,289 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:33,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8de7ceba3e95ef9d71ef663e0063791d: Waiting for close lock at 1676127633220Running coprocessor pre-close hooks at 1676127633220Disabling compacts and flushes for region at 1676127633220Disabling writes for close at 1676127633221 (+1 ms)Obtaining lock to block concurrent updates at 1676127633221Preparing flush snapshotting stores in 8de7ceba3e95ef9d71ef663e0063791d at 1676127633221Finished memstore snapshotting testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d., syncing WAL and waiting on mvcc, flushsize=dataSize=22709, getHeapSize=49056, getOffHeapSize=0, getCellsCount=339 at 1676127633222 (+1 ms)Flushing stores of testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. at 1676127633223 (+1 ms)Flushing 8de7ceba3e95ef9d71ef663e0063791d/cf: creating writer at 1676127633224 (+1 ms)Flushing mob file 8de7ceba3e95ef9d71ef663e0063791d/cf: appending metadata at 1676127633247 (+23 ms)Flushing mob file 8de7ceba3e95ef9d71ef663e0063791d/cf: closing flushed file at 1676127633247Flushing 8de7ceba3e95ef9d71ef663e0063791d/cf: appending metadata at 1676127633262 (+15 ms)Flushing 8de7ceba3e95ef9d71ef663e0063791d/cf: closing flushed file at 1676127633262Flushing 8de7ceba3e95ef9d71ef663e0063791d/cf: reopening flushed file at 1676127633280 (+18 ms)Finished flush of dataSize ~22.18 KB/22709, heapSize ~47.91 KB/49056, currentSize=0 B/0 for 8de7ceba3e95ef9d71ef663e0063791d in 63ms, sequenceid=8, compaction requested=false at 1676127633284 (+4 ms)Writing region close event to WAL at 1676127633286 (+2 ms)Running coprocessor post-close hooks at 1676127633289 (+3 ms)Closed at 1676127633289 2023-02-11 15:00:33,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:33,290 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=CLOSED 2023-02-11 15:00:33,290 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127633290"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127633290"}]},"ts":"1676127633290"} 2023-02-11 15:00:33,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=246 2023-02-11 15:00:33,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=246, state=SUCCESS; CloseRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785 in 268 msec 2023-02-11 15:00:33,292 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, UNASSIGN in 275 msec 2023-02-11 15:00:33,292 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=248 2023-02-11 15:00:33,292 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=248, state=SUCCESS; CloseRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743 in 271 msec 2023-02-11 15:00:33,293 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=248, resume processing ppid=242 2023-02-11 15:00:33,293 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, UNASSIGN in 276 msec 2023-02-11 15:00:33,294 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127633294"}]},"ts":"1676127633294"} 2023-02-11 15:00:33,295 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=DISABLED in hbase:meta 2023-02-11 15:00:33,296 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676127630926 to state=DISABLED 2023-02-11 15:00:33,297 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676127630926 in 286 msec 2023-02-11 15:00:33,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 15:00:33,765 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676127630926, procId: 242 completed 2023-02-11 15:00:33,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 } 2023-02-11 15:00:33,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676127633766 (current time:1676127633766). 2023-02-11 15:00:33,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 15:00:33,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(139): Connect 0x16ac890f to 127.0.0.1:58940 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 15:00:33,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c970782, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 15:00:33,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 15:00:33,776 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43202, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 15:00:33,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16ac890f to 127.0.0.1:58940 2023-02-11 15:00:33,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:33,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 15:00:33,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 } 2023-02-11 15:00:33,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676127630926, type=EXCLUSIVE 2023-02-11 15:00:33,779 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676127630926, type=EXCLUSIVE 2023-02-11 15:00:33,780 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676127630926, type=EXCLUSIVE; timeout=600000, timestamp=1676128233780 2023-02-11 15:00:33,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 } 2023-02-11 15:00:33,780 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676127630926 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676127630926 2023-02-11 15:00:33,780 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 15:00:33,780 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 15:00:33,788 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=DISABLED ttl=0 } 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:33,797 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 15:00:33,799 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/098b7a453bed4669b163aa8c746c8c95] hfiles 2023-02-11 15:00:33,799 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/098b7a453bed4669b163aa8c746c8c95 2023-02-11 15:00:33,800 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2d2aeada4c3d44b2a6b3e855d8a6a262] hfiles 2023-02-11 15:00:33,800 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2d2aeada4c3d44b2a6b3e855d8a6a262 2023-02-11 15:00:33,800 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/7ae84fa17bc340af8afaef45c904e668] hfiles 2023-02-11 15:00:33,800 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/7ae84fa17bc340af8afaef45c904e668 2023-02-11 15:00:33,800 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/4e63262457624e1c89e80e00eb7444c1] hfiles 2023-02-11 15:00:33,800 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/4e63262457624e1c89e80e00eb7444c1 2023-02-11 15:00:33,800 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf] hfiles 2023-02-11 15:00:33,800 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/708f88c8016544edb7fe2ac974bc6a97] hfiles 2023-02-11 15:00:33,800 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf 2023-02-11 15:00:33,800 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/708f88c8016544edb7fe2ac974bc6a97 2023-02-11 15:00:33,804 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/a87ff679a2f3e71d9181a67b7542122c202302110a80c9c9d4ca41218546e28ae6398848, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c4ca4238a0b923820dcc509a6f75849b20230211c776e56342844884b1e403356e08725c, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021183f238ac4534473e881d6ebe70c1e423, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/d41d8cd98f00b204e9800998ecf8427e202302118aa224b08bea443c9064e5c096ba8141, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/e4da3b7fbbce2345d7772b0674a318d520230211c1ae9b317a04417999a6bf1fa35fe905, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211049aa6a59ea446b7bd306f22bffc8da1] hfiles 2023-02-11 15:00:33,804 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/a87ff679a2f3e71d9181a67b7542122c202302110a80c9c9d4ca41218546e28ae6398848 2023-02-11 15:00:33,804 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c4ca4238a0b923820dcc509a6f75849b20230211c776e56342844884b1e403356e08725c 2023-02-11 15:00:33,804 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021183f238ac4534473e881d6ebe70c1e423 2023-02-11 15:00:33,804 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/d41d8cd98f00b204e9800998ecf8427e202302118aa224b08bea443c9064e5c096ba8141 2023-02-11 15:00:33,804 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/e4da3b7fbbce2345d7772b0674a318d520230211c1ae9b317a04417999a6bf1fa35fe905 2023-02-11 15:00:33,804 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211049aa6a59ea446b7bd306f22bffc8da1 2023-02-11 15:00:34,030 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 15:00:34,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 } is done 2023-02-11 15:00:34,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 }' is still in progress! 2023-02-11 15:00:34,032 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-11 15:00:34,223 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=DISABLED ttl=0 } as finished. 2023-02-11 15:00:34,223 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676127630926 2023-02-11 15:00:34,224 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb0-1676127630926 2023-02-11 15:00:34,243 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb0-1676127630926 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/snaptb0-1676127630926 2023-02-11 15:00:34,259 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676127630926 of table testCorruptedSnapshot-1676127630926 completed 2023-02-11 15:00:34,259 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb0-1676127630926 2023-02-11 15:00:34,259 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.hbase-snapshot/.tmp/snaptb0-1676127630926 2023-02-11 15:00:34,259 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676127630926 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676127630926 at 1676127633780Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=DISABLED ttl=0 } at 1676127633788 (+8 ms)Consolidate snapshot: snaptb0-1676127630926 at 1676127634223 (+435 ms)Loading Region manifests for snaptb0-1676127630926 at 1676127634223Writing data manifest for snaptb0-1676127630926 at 1676127634226 (+3 ms)Verifying snapshot: snaptb0-1676127630926 at 1676127634234 (+8 ms)Snapshot snaptb0-1676127630926 of table testCorruptedSnapshot-1676127630926 completed at 1676127634259 (+25 ms) 2023-02-11 15:00:34,260 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676127630926, type=EXCLUSIVE 2023-02-11 15:00:34,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676127630926, type=EXCLUSIVE in 482 msec 2023-02-11 15:00:34,532 DEBUG [Listener at localhost/39605] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 15:00:34,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 } is done 2023-02-11 15:00:34,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676127630926 table=testCorruptedSnapshot-1676127630926 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 15:00:34,533 INFO [Listener at localhost/39605] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676127630926 2023-02-11 15:00:34,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676127630926 2023-02-11 15:00:34,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:34,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 15:00:34,537 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676127630926 2023-02-11 15:00:34,537 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127634537"}]},"ts":"1676127634537"} 2023-02-11 15:00:34,538 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=ENABLING in hbase:meta 2023-02-11 15:00:34,540 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 15:00:34,540 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN}] 2023-02-11 15:00:34,541 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN 2023-02-11 15:00:34,541 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN 2023-02-11 15:00:34,541 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN 2023-02-11 15:00:34,541 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN 2023-02-11 15:00:34,541 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN 2023-02-11 15:00:34,542 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:34,542 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:34,542 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=true 2023-02-11 15:00:34,542 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=true 2023-02-11 15:00:34,542 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:34,542 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=true 2023-02-11 15:00:34,542 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:34,542 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:34,543 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33165,1676127604743; forceNewPlan=false, retain=true 2023-02-11 15:00:34,543 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN 2023-02-11 15:00:34,543 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42505,1676127604785; forceNewPlan=false, retain=true 2023-02-11 15:00:34,546 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:34,546 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41207,1676127604675; forceNewPlan=false, retain=true 2023-02-11 15:00:34,692 INFO [jenkins-hbase4:38517] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 15:00:34,692 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:34,692 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:34,692 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:34,693 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127634692"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127634692"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127634692"}]},"ts":"1676127634692"} 2023-02-11 15:00:34,693 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127634692"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127634692"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127634692"}]},"ts":"1676127634692"} 2023-02-11 15:00:34,692 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:34,692 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:34,693 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127634692"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127634692"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127634692"}]},"ts":"1676127634692"} 2023-02-11 15:00:34,693 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127634692"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127634692"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127634692"}]},"ts":"1676127634692"} 2023-02-11 15:00:34,693 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127634692"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127634692"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127634692"}]},"ts":"1676127634692"} 2023-02-11 15:00:34,694 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=257, state=RUNNABLE; OpenRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:34,695 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=260, state=RUNNABLE; OpenRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:34,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=259, state=RUNNABLE; OpenRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:34,697 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=261, state=RUNNABLE; OpenRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:34,698 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=262, state=RUNNABLE; OpenRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:34,698 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:34,698 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127634698"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127634698"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127634698"}]},"ts":"1676127634698"} 2023-02-11 15:00:34,700 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=258, state=RUNNABLE; OpenRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:34,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 15:00:34,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:34,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 238478a8a495efd2d422ce3a8631bbc2, NAME => 'testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 15:00:34,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:34,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:34,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:34,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:34,850 INFO [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:34,850 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:34,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 43b897ff4788f8ff736d623d593d53f4, NAME => 'testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 15:00:34,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:34,851 DEBUG [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf 2023-02-11 15:00:34,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:34,851 DEBUG [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf 2023-02-11 15:00:34,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:34,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:34,851 INFO [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 238478a8a495efd2d422ce3a8631bbc2 columnFamilyName cf 2023-02-11 15:00:34,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:34,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e7f1f013dd1caf6b5254f795ef0a3ed7, NAME => 'testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 15:00:34,852 INFO [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:34,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:34,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:34,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:34,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:34,852 DEBUG [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf 2023-02-11 15:00:34,853 DEBUG [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf 2023-02-11 15:00:34,853 INFO [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 43b897ff4788f8ff736d623d593d53f4 columnFamilyName cf 2023-02-11 15:00:34,853 INFO [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:34,854 DEBUG [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf 2023-02-11 15:00:34,854 DEBUG [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf 2023-02-11 15:00:34,854 INFO [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7f1f013dd1caf6b5254f795ef0a3ed7 columnFamilyName cf 2023-02-11 15:00:34,855 DEBUG [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf 2023-02-11 15:00:34,855 INFO [StoreOpener-238478a8a495efd2d422ce3a8631bbc2-1] regionserver.HStore(310): Store=238478a8a495efd2d422ce3a8631bbc2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:34,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:34,857 DEBUG [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/4e63262457624e1c89e80e00eb7444c1 2023-02-11 15:00:34,857 INFO [StoreOpener-43b897ff4788f8ff736d623d593d53f4-1] regionserver.HStore(310): Store=43b897ff4788f8ff736d623d593d53f4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:34,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:34,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:34,857 DEBUG [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/708f88c8016544edb7fe2ac974bc6a97 2023-02-11 15:00:34,858 INFO [StoreOpener-e7f1f013dd1caf6b5254f795ef0a3ed7-1] regionserver.HStore(310): Store=e7f1f013dd1caf6b5254f795ef0a3ed7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:34,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:34,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:34,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:34,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:34,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 238478a8a495efd2d422ce3a8631bbc2; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73618259, jitterRate=0.09699754416942596}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:34,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 238478a8a495efd2d422ce3a8631bbc2: Running coprocessor pre-open hook at 1676127634849Writing region info on filesystem at 1676127634849Initializing all the Stores at 1676127634849Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127634850 (+1 ms)Cleaning up temporary data from old regions at 1676127634857 (+7 ms)Cleaning up detritus from prior splits at 1676127634858 (+1 ms)Running coprocessor post-open hooks at 1676127634860 (+2 ms)Region opened successfully at 1676127634860 2023-02-11 15:00:34,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:34,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2., pid=263, masterSystemTime=1676127634846 2023-02-11 15:00:34,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 43b897ff4788f8ff736d623d593d53f4; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60694207, jitterRate=-0.09558583796024323}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:34,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 43b897ff4788f8ff736d623d593d53f4: Running coprocessor pre-open hook at 1676127634851Writing region info on filesystem at 1676127634851Initializing all the Stores at 1676127634851Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127634851Cleaning up temporary data from old regions at 1676127634859 (+8 ms)Cleaning up detritus from prior splits at 1676127634859Running coprocessor post-open hooks at 1676127634862 (+3 ms)Region opened successfully at 1676127634862 2023-02-11 15:00:34,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:34,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4., pid=264, masterSystemTime=1676127634848 2023-02-11 15:00:34,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e7f1f013dd1caf6b5254f795ef0a3ed7; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62003506, jitterRate=-0.07607576251029968}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:34,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e7f1f013dd1caf6b5254f795ef0a3ed7: Running coprocessor pre-open hook at 1676127634852Writing region info on filesystem at 1676127634852Initializing all the Stores at 1676127634853 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127634853Cleaning up temporary data from old regions at 1676127634859 (+6 ms)Cleaning up detritus from prior splits at 1676127634860 (+1 ms)Running coprocessor post-open hooks at 1676127634863 (+3 ms)Region opened successfully at 1676127634863 2023-02-11 15:00:34,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7., pid=268, masterSystemTime=1676127634850 2023-02-11 15:00:34,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:34,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:34,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:34,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 08d6264e7bb30445380a92d3761a3474, NAME => 'testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 15:00:34,865 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:34,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:34,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:34,865 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127634864"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127634864"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127634864"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127634864"}]},"ts":"1676127634864"} 2023-02-11 15:00:34,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:34,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:34,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:34,865 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:34,865 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:34,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a25d52f50b583b8024a28335f497bb9c, NAME => 'testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 15:00:34,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:34,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:34,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:34,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:34,866 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:34,866 INFO [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:34,866 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127634866"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127634866"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127634866"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127634866"}]},"ts":"1676127634866"} 2023-02-11 15:00:34,867 INFO [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:34,867 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:34,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:34,867 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127634867"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127634867"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127634867"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127634867"}]},"ts":"1676127634867"} 2023-02-11 15:00:34,867 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:34,867 DEBUG [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf 2023-02-11 15:00:34,867 DEBUG [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf 2023-02-11 15:00:34,867 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:34,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8de7ceba3e95ef9d71ef663e0063791d, NAME => 'testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 15:00:34,868 INFO [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08d6264e7bb30445380a92d3761a3474 columnFamilyName cf 2023-02-11 15:00:34,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676127630926 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:34,868 DEBUG [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf 2023-02-11 15:00:34,868 DEBUG [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf 2023-02-11 15:00:34,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 15:00:34,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:34,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:34,868 INFO [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a25d52f50b583b8024a28335f497bb9c columnFamilyName cf 2023-02-11 15:00:34,869 INFO [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:34,869 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=257 2023-02-11 15:00:34,870 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=257, state=SUCCESS; OpenRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675 in 172 msec 2023-02-11 15:00:34,871 DEBUG [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf 2023-02-11 15:00:34,871 DEBUG [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf 2023-02-11 15:00:34,871 INFO [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8de7ceba3e95ef9d71ef663e0063791d columnFamilyName cf 2023-02-11 15:00:34,871 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, ASSIGN in 329 msec 2023-02-11 15:00:34,871 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=260 2023-02-11 15:00:34,871 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=260, state=SUCCESS; OpenRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785 in 174 msec 2023-02-11 15:00:34,872 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=258 2023-02-11 15:00:34,872 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=258, state=SUCCESS; OpenRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743 in 170 msec 2023-02-11 15:00:34,873 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, ASSIGN in 331 msec 2023-02-11 15:00:34,873 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, ASSIGN in 332 msec 2023-02-11 15:00:34,878 DEBUG [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2d2aeada4c3d44b2a6b3e855d8a6a262 2023-02-11 15:00:34,880 INFO [StoreOpener-08d6264e7bb30445380a92d3761a3474-1] regionserver.HStore(310): Store=08d6264e7bb30445380a92d3761a3474/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:34,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:34,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:34,882 DEBUG [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/7ae84fa17bc340af8afaef45c904e668 2023-02-11 15:00:34,882 INFO [StoreOpener-a25d52f50b583b8024a28335f497bb9c-1] regionserver.HStore(310): Store=a25d52f50b583b8024a28335f497bb9c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:34,882 DEBUG [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] regionserver.HStore(539): loaded hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/098b7a453bed4669b163aa8c746c8c95 2023-02-11 15:00:34,882 INFO [StoreOpener-8de7ceba3e95ef9d71ef663e0063791d-1] regionserver.HStore(310): Store=8de7ceba3e95ef9d71ef663e0063791d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 15:00:34,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:34,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:34,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:34,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:34,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:34,885 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 08d6264e7bb30445380a92d3761a3474; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60976712, jitterRate=-0.09137618541717529}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:34,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 08d6264e7bb30445380a92d3761a3474: Running coprocessor pre-open hook at 1676127634865Writing region info on filesystem at 1676127634865Initializing all the Stores at 1676127634866 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127634866Cleaning up temporary data from old regions at 1676127634882 (+16 ms)Cleaning up detritus from prior splits at 1676127634882Running coprocessor post-open hooks at 1676127634885 (+3 ms)Region opened successfully at 1676127634885 2023-02-11 15:00:34,886 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474., pid=266, masterSystemTime=1676127634846 2023-02-11 15:00:34,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:34,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:34,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:34,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:34,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 8de7ceba3e95ef9d71ef663e0063791d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71809306, jitterRate=0.07004204392433167}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:34,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 8de7ceba3e95ef9d71ef663e0063791d: Running coprocessor pre-open hook at 1676127634868Writing region info on filesystem at 1676127634868Initializing all the Stores at 1676127634869 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127634869Cleaning up temporary data from old regions at 1676127634884 (+15 ms)Cleaning up detritus from prior splits at 1676127634885 (+1 ms)Running coprocessor post-open hooks at 1676127634888 (+3 ms)Region opened successfully at 1676127634888 2023-02-11 15:00:34,888 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:34,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a25d52f50b583b8024a28335f497bb9c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75470499, jitterRate=0.12459807097911835}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 15:00:34,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a25d52f50b583b8024a28335f497bb9c: Running coprocessor pre-open hook at 1676127634866Writing region info on filesystem at 1676127634866Initializing all the Stores at 1676127634866Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676127634867 (+1 ms)Cleaning up temporary data from old regions at 1676127634884 (+17 ms)Cleaning up detritus from prior splits at 1676127634885 (+1 ms)Running coprocessor post-open hooks at 1676127634888 (+3 ms)Region opened successfully at 1676127634888 2023-02-11 15:00:34,888 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127634888"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127634888"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127634888"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127634888"}]},"ts":"1676127634888"} 2023-02-11 15:00:34,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d., pid=267, masterSystemTime=1676127634850 2023-02-11 15:00:34,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c., pid=265, masterSystemTime=1676127634848 2023-02-11 15:00:34,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:34,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:34,890 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:34,890 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127634890"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127634890"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127634890"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127634890"}]},"ts":"1676127634890"} 2023-02-11 15:00:34,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=261 2023-02-11 15:00:34,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=261, state=SUCCESS; OpenRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675 in 192 msec 2023-02-11 15:00:34,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:34,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:34,891 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:34,891 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127634891"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676127634891"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676127634891"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676127634891"}]},"ts":"1676127634891"} 2023-02-11 15:00:34,892 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, ASSIGN in 351 msec 2023-02-11 15:00:34,893 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=259 2023-02-11 15:00:34,893 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=259, state=SUCCESS; OpenRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785 in 196 msec 2023-02-11 15:00:34,894 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=262 2023-02-11 15:00:34,894 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=262, state=SUCCESS; OpenRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743 in 195 msec 2023-02-11 15:00:34,894 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, ASSIGN in 353 msec 2023-02-11 15:00:34,895 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=262, resume processing ppid=256 2023-02-11 15:00:34,895 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, ASSIGN in 354 msec 2023-02-11 15:00:34,896 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127634896"}]},"ts":"1676127634896"} 2023-02-11 15:00:34,897 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=ENABLED in hbase:meta 2023-02-11 15:00:34,899 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676127630926' was successfully enabled. 2023-02-11 15:00:34,900 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676127630926 in 366 msec 2023-02-11 15:00:35,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 15:00:35,288 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676127630926, procId: 256 completed 2023-02-11 15:00:35,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:35,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:35,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:35,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:35,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41207] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:35,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33165] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 15:00:35,299 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676127630926 2023-02-11 15:00:35,299 DEBUG [Listener at localhost/39605] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:35,299 INFO [Listener at localhost/39605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 15:00:35,399 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 15:00:35,460 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 15:00:35,516 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 15:00:35,604 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ff51970 to 127.0.0.1:58940 2023-02-11 15:00:35,604 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:35,632 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 15:00:35,719 DEBUG [Listener at localhost/39605] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 15:00:36,545 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: 3ab632ae9ee54eca8e155e9f3ce8f4bf in the real (hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf) or archive (hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf) directory for the primary table. 2023-02-11 15:00:36,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 3ab632ae9ee54eca8e155e9f3ce8f4bf in the real (hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf) or archive (hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 15:00:36,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] ipc.CallRunner(144): callId: 358 service: MasterService methodName: RestoreSnapshot size: 114 connection: 172.31.14.131:41390 deadline: 1676127696541, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 3ab632ae9ee54eca8e155e9f3ce8f4bf in the real (hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf) or archive (hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/3ab632ae9ee54eca8e155e9f3ce8f4bf) directory for the primary table. 2023-02-11 15:00:36,565 INFO [Listener at localhost/39605] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676127630926 2023-02-11 15:00:36,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676127630926 2023-02-11 15:00:36,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:36,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 15:00:36,569 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127636569"}]},"ts":"1676127636569"} 2023-02-11 15:00:36,570 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=DISABLING in hbase:meta 2023-02-11 15:00:36,576 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676127630926 to state=DISABLING 2023-02-11 15:00:36,577 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, UNASSIGN}] 2023-02-11 15:00:36,578 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, UNASSIGN 2023-02-11 15:00:36,578 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, UNASSIGN 2023-02-11 15:00:36,578 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, UNASSIGN 2023-02-11 15:00:36,578 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, UNASSIGN 2023-02-11 15:00:36,578 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, UNASSIGN 2023-02-11 15:00:36,578 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:36,578 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127636578"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127636578"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127636578"}]},"ts":"1676127636578"} 2023-02-11 15:00:36,578 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:36,579 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:36,579 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127636578"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127636578"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127636578"}]},"ts":"1676127636578"} 2023-02-11 15:00:36,579 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:36,578 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:36,579 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127636579"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127636579"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127636579"}]},"ts":"1676127636579"} 2023-02-11 15:00:36,579 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127636578"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127636578"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127636578"}]},"ts":"1676127636578"} 2023-02-11 15:00:36,579 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127636579"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127636579"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127636579"}]},"ts":"1676127636579"} 2023-02-11 15:00:36,580 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=275, state=RUNNABLE; CloseRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:36,580 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=273, state=RUNNABLE; CloseRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:36,581 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=276, state=RUNNABLE; CloseRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:36,581 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=274, state=RUNNABLE; CloseRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785}] 2023-02-11 15:00:36,582 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, UNASSIGN 2023-02-11 15:00:36,582 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=272, state=RUNNABLE; CloseRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743}] 2023-02-11 15:00:36,582 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:36,582 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127636582"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676127636582"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676127636582"}]},"ts":"1676127636582"} 2023-02-11 15:00:36,583 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675}] 2023-02-11 15:00:36,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:36,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:36,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a25d52f50b583b8024a28335f497bb9c, disabling compactions & flushes 2023-02-11 15:00:36,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 238478a8a495efd2d422ce3a8631bbc2, disabling compactions & flushes 2023-02-11 15:00:36,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:36,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:36,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:36,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:36,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. after waiting 0 ms 2023-02-11 15:00:36,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. after waiting 0 ms 2023-02-11 15:00:36,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:36,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:36,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 238478a8a495efd2d422ce3a8631bbc2 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-11 15:00:36,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a25d52f50b583b8024a28335f497bb9c 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-11 15:00:36,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:36,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 8de7ceba3e95ef9d71ef663e0063791d, disabling compactions & flushes 2023-02-11 15:00:36,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:36,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:36,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. after waiting 0 ms 2023-02-11 15:00:36,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:36,733 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 8de7ceba3e95ef9d71ef663e0063791d 1/1 column families, dataSize=22.11 KB heapSize=47.78 KB 2023-02-11 15:00:36,750 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230211543f42fd7f0c43f38d04cf974eb54ac3 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c81e728d9d4c2f636f067f89cc14862c20230211543f42fd7f0c43f38d04cf974eb54ac3 2023-02-11 15:00:36,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023021117629058fe214714ae7dd4c3795b0739 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/d41d8cd98f00b204e9800998ecf8427e2023021117629058fe214714ae7dd4c3795b0739 2023-02-11 15:00:36,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021113eac0ece01f41228b7d97c0a18e5d8d to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/e4da3b7fbbce2345d7772b0674a318d52023021113eac0ece01f41228b7d97c0a18e5d8d 2023-02-11 15:00:36,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/.tmp/cf/f859861c9a494207bbd5638184604e92 2023-02-11 15:00:36,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/.tmp/cf/0694f77621ee401f935799b1c65ca208 2023-02-11 15:00:36,766 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/.tmp/cf/f859861c9a494207bbd5638184604e92 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/f859861c9a494207bbd5638184604e92 2023-02-11 15:00:36,767 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/.tmp/cf/0694f77621ee401f935799b1c65ca208 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/0694f77621ee401f935799b1c65ca208 2023-02-11 15:00:36,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/f859861c9a494207bbd5638184604e92, entries=27, sequenceid=15, filesize=9.6 K 2023-02-11 15:00:36,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/0694f77621ee401f935799b1c65ca208, entries=39, sequenceid=15, filesize=11.6 K 2023-02-11 15:00:36,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for a25d52f50b583b8024a28335f497bb9c in 38ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:36,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 238478a8a495efd2d422ce3a8631bbc2 in 41ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:36,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:36,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. 2023-02-11 15:00:36,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a25d52f50b583b8024a28335f497bb9c: Waiting for close lock at 1676127636731Running coprocessor pre-close hooks at 1676127636731Disabling compacts and flushes for region at 1676127636731Disabling writes for close at 1676127636732 (+1 ms)Obtaining lock to block concurrent updates at 1676127636732Preparing flush snapshotting stores in a25d52f50b583b8024a28335f497bb9c at 1676127636732Finished memstore snapshotting testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676127636732Flushing stores of testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c. at 1676127636732Flushing a25d52f50b583b8024a28335f497bb9c/cf: creating writer at 1676127636734 (+2 ms)Flushing mob file a25d52f50b583b8024a28335f497bb9c/cf: appending metadata at 1676127636738 (+4 ms)Flushing mob file a25d52f50b583b8024a28335f497bb9c/cf: closing flushed file at 1676127636738Flushing a25d52f50b583b8024a28335f497bb9c/cf: appending metadata at 1676127636751 (+13 ms)Flushing a25d52f50b583b8024a28335f497bb9c/cf: closing flushed file at 1676127636751Flushing a25d52f50b583b8024a28335f497bb9c/cf: reopening flushed file at 1676127636766 (+15 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for a25d52f50b583b8024a28335f497bb9c in 38ms, sequenceid=15, compaction requested=false at 1676127636770 (+4 ms)Writing region close event to WAL at 1676127636770Running coprocessor post-close hooks at 1676127636777 (+7 ms)Closed at 1676127636777 2023-02-11 15:00:36,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:36,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:36,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 43b897ff4788f8ff736d623d593d53f4, disabling compactions & flushes 2023-02-11 15:00:36,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:36,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:36,781 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=a25d52f50b583b8024a28335f497bb9c, regionState=CLOSED 2023-02-11 15:00:36,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. after waiting 0 ms 2023-02-11 15:00:36,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:36,782 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127636781"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127636781"}]},"ts":"1676127636781"} 2023-02-11 15:00:36,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 43b897ff4788f8ff736d623d593d53f4 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-11 15:00:36,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:36,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. 2023-02-11 15:00:36,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 238478a8a495efd2d422ce3a8631bbc2: Waiting for close lock at 1676127636731Running coprocessor pre-close hooks at 1676127636731Disabling compacts and flushes for region at 1676127636731Disabling writes for close at 1676127636732 (+1 ms)Obtaining lock to block concurrent updates at 1676127636732Preparing flush snapshotting stores in 238478a8a495efd2d422ce3a8631bbc2 at 1676127636732Finished memstore snapshotting testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676127636732Flushing stores of testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2. at 1676127636733 (+1 ms)Flushing 238478a8a495efd2d422ce3a8631bbc2/cf: creating writer at 1676127636734 (+1 ms)Flushing mob file 238478a8a495efd2d422ce3a8631bbc2/cf: appending metadata at 1676127636740 (+6 ms)Flushing mob file 238478a8a495efd2d422ce3a8631bbc2/cf: closing flushed file at 1676127636740Flushing 238478a8a495efd2d422ce3a8631bbc2/cf: appending metadata at 1676127636752 (+12 ms)Flushing 238478a8a495efd2d422ce3a8631bbc2/cf: closing flushed file at 1676127636752Flushing 238478a8a495efd2d422ce3a8631bbc2/cf: reopening flushed file at 1676127636767 (+15 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 238478a8a495efd2d422ce3a8631bbc2 in 41ms, sequenceid=15, compaction requested=false at 1676127636773 (+6 ms)Writing region close event to WAL at 1676127636776 (+3 ms)Running coprocessor post-close hooks at 1676127636784 (+8 ms)Closed at 1676127636784 2023-02-11 15:00:36,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:36,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:36,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 08d6264e7bb30445380a92d3761a3474, disabling compactions & flushes 2023-02-11 15:00:36,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:36,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:36,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. after waiting 0 ms 2023-02-11 15:00:36,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:36,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 08d6264e7bb30445380a92d3761a3474 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-11 15:00:36,787 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=273 2023-02-11 15:00:36,787 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=238478a8a495efd2d422ce3a8631bbc2, regionState=CLOSED 2023-02-11 15:00:36,787 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=273, state=SUCCESS; CloseRegionProcedure a25d52f50b583b8024a28335f497bb9c, server=jenkins-hbase4.apache.org,42505,1676127604785 in 204 msec 2023-02-11 15:00:36,787 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127636787"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127636787"}]},"ts":"1676127636787"} 2023-02-11 15:00:36,788 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=a25d52f50b583b8024a28335f497bb9c, UNASSIGN in 210 msec 2023-02-11 15:00:36,790 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-02-11 15:00:36,790 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure 238478a8a495efd2d422ce3a8631bbc2, server=jenkins-hbase4.apache.org,41207,1676127604675 in 206 msec 2023-02-11 15:00:36,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=238478a8a495efd2d422ce3a8631bbc2, UNASSIGN in 213 msec 2023-02-11 15:00:36,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021178bff0d94eb34671bb15916ed514cec5 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021178bff0d94eb34671bb15916ed514cec5 2023-02-11 15:00:36,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302110dba2ebeb3bf45f3851dbda8a898b7fa to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/a87ff679a2f3e71d9181a67b7542122c202302110dba2ebeb3bf45f3851dbda8a898b7fa 2023-02-11 15:00:36,805 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/.tmp/cf/dbaef4efa81a48c69fbb70394d1f0843 2023-02-11 15:00:36,808 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/.tmp/cf/2409bd3c5f3840bc9bee7b0f0ddaa4fe 2023-02-11 15:00:36,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/.tmp/cf/dbaef4efa81a48c69fbb70394d1f0843 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/dbaef4efa81a48c69fbb70394d1f0843 2023-02-11 15:00:36,811 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/.tmp/cf/2409bd3c5f3840bc9bee7b0f0ddaa4fe as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2409bd3c5f3840bc9bee7b0f0ddaa4fe 2023-02-11 15:00:36,811 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/dbaef4efa81a48c69fbb70394d1f0843, entries=36, sequenceid=15, filesize=11.1 K 2023-02-11 15:00:36,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 43b897ff4788f8ff736d623d593d53f4 in 30ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:36,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2409bd3c5f3840bc9bee7b0f0ddaa4fe, entries=31, sequenceid=15, filesize=10.2 K 2023-02-11 15:00:36,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 08d6264e7bb30445380a92d3761a3474 in 29ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:36,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:36,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. 2023-02-11 15:00:36,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 43b897ff4788f8ff736d623d593d53f4: Waiting for close lock at 1676127636781Running coprocessor pre-close hooks at 1676127636781Disabling compacts and flushes for region at 1676127636781Disabling writes for close at 1676127636782 (+1 ms)Obtaining lock to block concurrent updates at 1676127636782Preparing flush snapshotting stores in 43b897ff4788f8ff736d623d593d53f4 at 1676127636782Finished memstore snapshotting testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676127636782Flushing stores of testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4. at 1676127636783 (+1 ms)Flushing 43b897ff4788f8ff736d623d593d53f4/cf: creating writer at 1676127636784 (+1 ms)Flushing mob file 43b897ff4788f8ff736d623d593d53f4/cf: appending metadata at 1676127636788 (+4 ms)Flushing mob file 43b897ff4788f8ff736d623d593d53f4/cf: closing flushed file at 1676127636788Flushing 43b897ff4788f8ff736d623d593d53f4/cf: appending metadata at 1676127636799 (+11 ms)Flushing 43b897ff4788f8ff736d623d593d53f4/cf: closing flushed file at 1676127636799Flushing 43b897ff4788f8ff736d623d593d53f4/cf: reopening flushed file at 1676127636809 (+10 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 43b897ff4788f8ff736d623d593d53f4 in 30ms, sequenceid=15, compaction requested=false at 1676127636812 (+3 ms)Writing region close event to WAL at 1676127636813 (+1 ms)Running coprocessor post-close hooks at 1676127636815 (+2 ms)Closed at 1676127636815 2023-02-11 15:00:36,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:36,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:36,817 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=43b897ff4788f8ff736d623d593d53f4, regionState=CLOSED 2023-02-11 15:00:36,817 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127636817"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127636817"}]},"ts":"1676127636817"} 2023-02-11 15:00:36,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. 2023-02-11 15:00:36,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 08d6264e7bb30445380a92d3761a3474: Waiting for close lock at 1676127636785Running coprocessor pre-close hooks at 1676127636785Disabling compacts and flushes for region at 1676127636785Disabling writes for close at 1676127636785Obtaining lock to block concurrent updates at 1676127636785Preparing flush snapshotting stores in 08d6264e7bb30445380a92d3761a3474 at 1676127636785Finished memstore snapshotting testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676127636786 (+1 ms)Flushing stores of testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474. at 1676127636786Flushing 08d6264e7bb30445380a92d3761a3474/cf: creating writer at 1676127636787 (+1 ms)Flushing mob file 08d6264e7bb30445380a92d3761a3474/cf: appending metadata at 1676127636792 (+5 ms)Flushing mob file 08d6264e7bb30445380a92d3761a3474/cf: closing flushed file at 1676127636792Flushing 08d6264e7bb30445380a92d3761a3474/cf: appending metadata at 1676127636801 (+9 ms)Flushing 08d6264e7bb30445380a92d3761a3474/cf: closing flushed file at 1676127636801Flushing 08d6264e7bb30445380a92d3761a3474/cf: reopening flushed file at 1676127636811 (+10 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 08d6264e7bb30445380a92d3761a3474 in 29ms, sequenceid=15, compaction requested=false at 1676127636814 (+3 ms)Writing region close event to WAL at 1676127636815 (+1 ms)Running coprocessor post-close hooks at 1676127636817 (+2 ms)Closed at 1676127636817 2023-02-11 15:00:36,819 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:36,819 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=08d6264e7bb30445380a92d3761a3474, regionState=CLOSED 2023-02-11 15:00:36,819 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127636819"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127636819"}]},"ts":"1676127636819"} 2023-02-11 15:00:36,820 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=274 2023-02-11 15:00:36,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 15:00:36,820 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=274, state=SUCCESS; CloseRegionProcedure 43b897ff4788f8ff736d623d593d53f4, server=jenkins-hbase4.apache.org,42505,1676127604785 in 237 msec 2023-02-11 15:00:36,821 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=43b897ff4788f8ff736d623d593d53f4, UNASSIGN in 243 msec 2023-02-11 15:00:36,821 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=275 2023-02-11 15:00:36,821 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=275, state=SUCCESS; CloseRegionProcedure 08d6264e7bb30445380a92d3761a3474, server=jenkins-hbase4.apache.org,41207,1676127604675 in 241 msec 2023-02-11 15:00:36,822 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=08d6264e7bb30445380a92d3761a3474, UNASSIGN in 244 msec 2023-02-11 15:00:37,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/.tmp/cf/230e33a2d2d340acb9c61d4ecf72fa53 2023-02-11 15:00:37,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/.tmp/cf/230e33a2d2d340acb9c61d4ecf72fa53 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/230e33a2d2d340acb9c61d4ecf72fa53 2023-02-11 15:00:37,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/230e33a2d2d340acb9c61d4ecf72fa53, entries=338, sequenceid=15, filesize=63.2 K 2023-02-11 15:00:37,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.11 KB/22642, heapSize ~47.77 KB/48912, currentSize=0 B/0 for 8de7ceba3e95ef9d71ef663e0063791d in 440ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:37,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:37,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. 2023-02-11 15:00:37,175 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 8de7ceba3e95ef9d71ef663e0063791d: Waiting for close lock at 1676127636732Running coprocessor pre-close hooks at 1676127636732Disabling compacts and flushes for region at 1676127636732Disabling writes for close at 1676127636732Obtaining lock to block concurrent updates at 1676127636734 (+2 ms)Preparing flush snapshotting stores in 8de7ceba3e95ef9d71ef663e0063791d at 1676127636734Finished memstore snapshotting testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d., syncing WAL and waiting on mvcc, flushsize=dataSize=22642, getHeapSize=48912, getOffHeapSize=0, getCellsCount=338 at 1676127636734Flushing stores of testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d. at 1676127636734Flushing 8de7ceba3e95ef9d71ef663e0063791d/cf: creating writer at 1676127636735 (+1 ms)Flushing mob file 8de7ceba3e95ef9d71ef663e0063791d/cf: appending metadata at 1676127636740 (+5 ms)Flushing mob file 8de7ceba3e95ef9d71ef663e0063791d/cf: closing flushed file at 1676127636740Flushing 8de7ceba3e95ef9d71ef663e0063791d/cf: appending metadata at 1676127636752 (+12 ms)Flushing 8de7ceba3e95ef9d71ef663e0063791d/cf: closing flushed file at 1676127636752Flushing 8de7ceba3e95ef9d71ef663e0063791d/cf: reopening flushed file at 1676127637169 (+417 ms)Finished flush of dataSize ~22.11 KB/22642, heapSize ~47.77 KB/48912, currentSize=0 B/0 for 8de7ceba3e95ef9d71ef663e0063791d in 440ms, sequenceid=15, compaction requested=false at 1676127637172 (+3 ms)Writing region close event to WAL at 1676127637172Running coprocessor post-close hooks at 1676127637175 (+3 ms)Closed at 1676127637175 2023-02-11 15:00:37,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:37,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e7f1f013dd1caf6b5254f795ef0a3ed7, disabling compactions & flushes 2023-02-11 15:00:37,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. after waiting 0 ms 2023-02-11 15:00:37,176 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=8de7ceba3e95ef9d71ef663e0063791d, regionState=CLOSED 2023-02-11 15:00:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:37,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e7f1f013dd1caf6b5254f795ef0a3ed7 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-11 15:00:37,177 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676127637176"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127637176"}]},"ts":"1676127637176"} 2023-02-11 15:00:37,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=276 2023-02-11 15:00:37,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=276, state=SUCCESS; CloseRegionProcedure 8de7ceba3e95ef9d71ef663e0063791d, server=jenkins-hbase4.apache.org,33165,1676127604743 in 597 msec 2023-02-11 15:00:37,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=8de7ceba3e95ef9d71ef663e0063791d, UNASSIGN in 603 msec 2023-02-11 15:00:37,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302113cf255632ab04b78bcc5103f532103f6 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c4ca4238a0b923820dcc509a6f75849b202302113cf255632ab04b78bcc5103f532103f6 2023-02-11 15:00:37,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/.tmp/cf/93601d162ce44ad38931e494f3779e3c 2023-02-11 15:00:37,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/.tmp/cf/93601d162ce44ad38931e494f3779e3c as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/93601d162ce44ad38931e494f3779e3c 2023-02-11 15:00:37,208 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/93601d162ce44ad38931e494f3779e3c, entries=29, sequenceid=15, filesize=9.9 K 2023-02-11 15:00:37,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for e7f1f013dd1caf6b5254f795ef0a3ed7 in 31ms, sequenceid=15, compaction requested=false 2023-02-11 15:00:37,211 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 15:00:37,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. 2023-02-11 15:00:37,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e7f1f013dd1caf6b5254f795ef0a3ed7: Waiting for close lock at 1676127637176Running coprocessor pre-close hooks at 1676127637176Disabling compacts and flushes for region at 1676127637176Disabling writes for close at 1676127637176Obtaining lock to block concurrent updates at 1676127637177 (+1 ms)Preparing flush snapshotting stores in e7f1f013dd1caf6b5254f795ef0a3ed7 at 1676127637177Finished memstore snapshotting testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676127637177Flushing stores of testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7. at 1676127637177Flushing e7f1f013dd1caf6b5254f795ef0a3ed7/cf: creating writer at 1676127637178 (+1 ms)Flushing mob file e7f1f013dd1caf6b5254f795ef0a3ed7/cf: appending metadata at 1676127637183 (+5 ms)Flushing mob file e7f1f013dd1caf6b5254f795ef0a3ed7/cf: closing flushed file at 1676127637183Flushing e7f1f013dd1caf6b5254f795ef0a3ed7/cf: appending metadata at 1676127637191 (+8 ms)Flushing e7f1f013dd1caf6b5254f795ef0a3ed7/cf: closing flushed file at 1676127637192 (+1 ms)Flushing e7f1f013dd1caf6b5254f795ef0a3ed7/cf: reopening flushed file at 1676127637206 (+14 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for e7f1f013dd1caf6b5254f795ef0a3ed7 in 31ms, sequenceid=15, compaction requested=false at 1676127637209 (+3 ms)Writing region close event to WAL at 1676127637209Running coprocessor post-close hooks at 1676127637211 (+2 ms)Closed at 1676127637211 2023-02-11 15:00:37,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:37,213 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=e7f1f013dd1caf6b5254f795ef0a3ed7, regionState=CLOSED 2023-02-11 15:00:37,213 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676127637213"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676127637213"}]},"ts":"1676127637213"} 2023-02-11 15:00:37,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=272 2023-02-11 15:00:37,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=272, state=SUCCESS; CloseRegionProcedure e7f1f013dd1caf6b5254f795ef0a3ed7, server=jenkins-hbase4.apache.org,33165,1676127604743 in 632 msec 2023-02-11 15:00:37,216 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=272, resume processing ppid=270 2023-02-11 15:00:37,217 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676127630926, region=e7f1f013dd1caf6b5254f795ef0a3ed7, UNASSIGN in 638 msec 2023-02-11 15:00:37,217 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676127637217"}]},"ts":"1676127637217"} 2023-02-11 15:00:37,218 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676127630926, state=DISABLED in hbase:meta 2023-02-11 15:00:37,219 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676127630926 to state=DISABLED 2023-02-11 15:00:37,221 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676127630926 in 654 msec 2023-02-11 15:00:37,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 15:00:37,321 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676127630926, procId: 270 completed 2023-02-11 15:00:37,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1676127630926 2023-02-11 15:00:37,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:37,324 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:37,325 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:37,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-11 15:00:37,332 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:37,332 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:37,332 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:37,332 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:37,332 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:37,332 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:37,334 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/recovered.edits] 2023-02-11 15:00:37,334 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/recovered.edits] 2023-02-11 15:00:37,334 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/recovered.edits] 2023-02-11 15:00:37,335 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/recovered.edits] 2023-02-11 15:00:37,335 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/recovered.edits] 2023-02-11 15:00:37,335 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf, FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/recovered.edits] 2023-02-11 15:00:37,340 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/4e63262457624e1c89e80e00eb7444c1 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/4e63262457624e1c89e80e00eb7444c1 2023-02-11 15:00:37,340 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/098b7a453bed4669b163aa8c746c8c95 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/098b7a453bed4669b163aa8c746c8c95 2023-02-11 15:00:37,340 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/7ae84fa17bc340af8afaef45c904e668 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/7ae84fa17bc340af8afaef45c904e668 2023-02-11 15:00:37,340 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/708f88c8016544edb7fe2ac974bc6a97 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/708f88c8016544edb7fe2ac974bc6a97 2023-02-11 15:00:37,340 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/0694f77621ee401f935799b1c65ca208 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/cf/0694f77621ee401f935799b1c65ca208 2023-02-11 15:00:37,340 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2409bd3c5f3840bc9bee7b0f0ddaa4fe to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2409bd3c5f3840bc9bee7b0f0ddaa4fe 2023-02-11 15:00:37,342 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/93601d162ce44ad38931e494f3779e3c to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/cf/93601d162ce44ad38931e494f3779e3c 2023-02-11 15:00:37,342 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/dbaef4efa81a48c69fbb70394d1f0843 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/cf/dbaef4efa81a48c69fbb70394d1f0843 2023-02-11 15:00:37,342 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/f859861c9a494207bbd5638184604e92 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/cf/f859861c9a494207bbd5638184604e92 2023-02-11 15:00:37,342 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/230e33a2d2d340acb9c61d4ecf72fa53 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/cf/230e33a2d2d340acb9c61d4ecf72fa53 2023-02-11 15:00:37,342 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2d2aeada4c3d44b2a6b3e855d8a6a262 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/cf/2d2aeada4c3d44b2a6b3e855d8a6a262 2023-02-11 15:00:37,345 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2/recovered.edits/18.seqid 2023-02-11 15:00:37,345 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c/recovered.edits/18.seqid 2023-02-11 15:00:37,346 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4/recovered.edits/18.seqid 2023-02-11 15:00:37,346 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/238478a8a495efd2d422ce3a8631bbc2 2023-02-11 15:00:37,346 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7/recovered.edits/18.seqid 2023-02-11 15:00:37,346 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/a25d52f50b583b8024a28335f497bb9c 2023-02-11 15:00:37,347 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474/recovered.edits/18.seqid 2023-02-11 15:00:37,347 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/recovered.edits/18.seqid to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d/recovered.edits/18.seqid 2023-02-11 15:00:37,347 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/43b897ff4788f8ff736d623d593d53f4 2023-02-11 15:00:37,347 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/e7f1f013dd1caf6b5254f795ef0a3ed7 2023-02-11 15:00:37,347 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/08d6264e7bb30445380a92d3761a3474 2023-02-11 15:00:37,348 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/.tmp/data/default/testCorruptedSnapshot-1676127630926/8de7ceba3e95ef9d71ef663e0063791d 2023-02-11 15:00:37,348 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676127630926 regions 2023-02-11 15:00:37,348 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6 2023-02-11 15:00:37,349 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf] 2023-02-11 15:00:37,351 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/a87ff679a2f3e71d9181a67b7542122c202302110a80c9c9d4ca41218546e28ae6398848 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/a87ff679a2f3e71d9181a67b7542122c202302110a80c9c9d4ca41218546e28ae6398848 2023-02-11 15:00:37,352 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/a87ff679a2f3e71d9181a67b7542122c202302110dba2ebeb3bf45f3851dbda8a898b7fa to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/a87ff679a2f3e71d9181a67b7542122c202302110dba2ebeb3bf45f3851dbda8a898b7fa 2023-02-11 15:00:37,353 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c4ca4238a0b923820dcc509a6f75849b202302113cf255632ab04b78bcc5103f532103f6 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c4ca4238a0b923820dcc509a6f75849b202302113cf255632ab04b78bcc5103f532103f6 2023-02-11 15:00:37,354 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c4ca4238a0b923820dcc509a6f75849b20230211c776e56342844884b1e403356e08725c to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c4ca4238a0b923820dcc509a6f75849b20230211c776e56342844884b1e403356e08725c 2023-02-11 15:00:37,355 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c81e728d9d4c2f636f067f89cc14862c20230211543f42fd7f0c43f38d04cf974eb54ac3 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c81e728d9d4c2f636f067f89cc14862c20230211543f42fd7f0c43f38d04cf974eb54ac3 2023-02-11 15:00:37,356 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021183f238ac4534473e881d6ebe70c1e423 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021183f238ac4534473e881d6ebe70c1e423 2023-02-11 15:00:37,357 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/d41d8cd98f00b204e9800998ecf8427e2023021117629058fe214714ae7dd4c3795b0739 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/d41d8cd98f00b204e9800998ecf8427e2023021117629058fe214714ae7dd4c3795b0739 2023-02-11 15:00:37,358 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/d41d8cd98f00b204e9800998ecf8427e202302118aa224b08bea443c9064e5c096ba8141 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/d41d8cd98f00b204e9800998ecf8427e202302118aa224b08bea443c9064e5c096ba8141 2023-02-11 15:00:37,358 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/e4da3b7fbbce2345d7772b0674a318d52023021113eac0ece01f41228b7d97c0a18e5d8d to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/e4da3b7fbbce2345d7772b0674a318d52023021113eac0ece01f41228b7d97c0a18e5d8d 2023-02-11 15:00:37,359 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/e4da3b7fbbce2345d7772b0674a318d520230211c1ae9b317a04417999a6bf1fa35fe905 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/e4da3b7fbbce2345d7772b0674a318d520230211c1ae9b317a04417999a6bf1fa35fe905 2023-02-11 15:00:37,360 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211049aa6a59ea446b7bd306f22bffc8da1 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211049aa6a59ea446b7bd306f22bffc8da1 2023-02-11 15:00:37,361 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021178bff0d94eb34671bb15916ed514cec5 to hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/archive/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021178bff0d94eb34671bb15916ed514cec5 2023-02-11 15:00:37,361 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/mobdir/data/default/testCorruptedSnapshot-1676127630926/624599ea0aa4d4991641f8f61bb7e1c6 2023-02-11 15:00:37,363 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:37,365 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676127630926 from hbase:meta 2023-02-11 15:00:37,367 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676127630926' descriptor. 2023-02-11 15:00:37,368 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:37,368 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676127630926' from region states. 2023-02-11 15:00:37,368 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127637368"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:37,368 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127637368"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:37,368 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127637368"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:37,368 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127637368"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:37,368 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127637368"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:37,368 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676127637368"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:37,370 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 15:00:37,370 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 238478a8a495efd2d422ce3a8631bbc2, NAME => 'testCorruptedSnapshot-1676127630926,,1676127630928.238478a8a495efd2d422ce3a8631bbc2.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e7f1f013dd1caf6b5254f795ef0a3ed7, NAME => 'testCorruptedSnapshot-1676127630926,1,1676127630928.e7f1f013dd1caf6b5254f795ef0a3ed7.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => a25d52f50b583b8024a28335f497bb9c, NAME => 'testCorruptedSnapshot-1676127630926,2,1676127630928.a25d52f50b583b8024a28335f497bb9c.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 43b897ff4788f8ff736d623d593d53f4, NAME => 'testCorruptedSnapshot-1676127630926,3,1676127630928.43b897ff4788f8ff736d623d593d53f4.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 08d6264e7bb30445380a92d3761a3474, NAME => 'testCorruptedSnapshot-1676127630926,4,1676127630928.08d6264e7bb30445380a92d3761a3474.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 8de7ceba3e95ef9d71ef663e0063791d, NAME => 'testCorruptedSnapshot-1676127630926,5,1676127630928.8de7ceba3e95ef9d71ef663e0063791d.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 15:00:37,370 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676127630926' as deleted. 2023-02-11 15:00:37,370 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676127630926","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676127637370"}]},"ts":"9223372036854775807"} 2023-02-11 15:00:37,371 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676127630926 state from META 2023-02-11 15:00:37,372 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 15:00:37,373 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676127630926 2023-02-11 15:00:37,374 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676127630926 in 51 msec 2023-02-11 15:00:37,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-11 15:00:37,577 INFO [Listener at localhost/39605] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676127630926, procId: 283 completed 2023-02-11 15:00:37,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676127630926" 2023-02-11 15:00:37,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676127630926 2023-02-11 15:00:37,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676127630926" 2023-02-11 15:00:37,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38517] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676127630926 2023-02-11 15:00:37,606 INFO [Listener at localhost/39605] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=422 (was 426), OpenFileDescriptor=796 (was 812), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=260 (was 283), ProcessCount=168 (was 168), AvailableMemoryMB=3211 (was 3243) 2023-02-11 15:00:37,606 INFO [Listener at localhost/39605] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-11 15:00:37,606 INFO [Listener at localhost/39605] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-11 15:00:37,608 DEBUG [Listener at localhost/39605] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c633018 to 127.0.0.1:58940 2023-02-11 15:00:37,608 DEBUG [Listener at localhost/39605] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:37,608 DEBUG [Listener at localhost/39605] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-11 15:00:37,608 DEBUG [Listener at localhost/39605] util.JVMClusterUtil(257): Found active master hash=1387558400, stopped=false 2023-02-11 15:00:37,608 INFO [Listener at localhost/39605] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,38517,1676127603486 2023-02-11 15:00:37,613 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 15:00:37,613 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 15:00:37,614 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 15:00:37,614 INFO [Listener at localhost/39605] procedure2.ProcedureExecutor(629): Stopping 2023-02-11 15:00:37,613 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 15:00:37,614 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 15:00:37,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 15:00:37,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 15:00:37,614 DEBUG [Listener at localhost/39605] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ef34e61 to 127.0.0.1:58940 2023-02-11 15:00:37,614 DEBUG [Listener at localhost/39605] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:37,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 15:00:37,615 INFO [Listener at localhost/39605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41207,1676127604675' ***** 2023-02-11 15:00:37,615 INFO [Listener at localhost/39605] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 15:00:37,615 INFO [Listener at localhost/39605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33165,1676127604743' ***** 2023-02-11 15:00:37,615 INFO [Listener at localhost/39605] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 15:00:37,615 INFO [Listener at localhost/39605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42505,1676127604785' ***** 2023-02-11 15:00:37,615 INFO [Listener at localhost/39605] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 15:00:37,615 INFO [RS:0;jenkins-hbase4:41207] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 15:00:37,615 INFO [RS:1;jenkins-hbase4:33165] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 15:00:37,615 INFO [RS:2;jenkins-hbase4:42505] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 15:00:37,615 INFO [RS:1;jenkins-hbase4:33165] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 15:00:37,615 INFO [RS:2;jenkins-hbase4:42505] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 15:00:37,615 INFO [RS:1;jenkins-hbase4:33165] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 15:00:37,615 INFO [RS:0;jenkins-hbase4:41207] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 15:00:37,615 INFO [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:37,616 INFO [RS:0;jenkins-hbase4:41207] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 15:00:37,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 15:00:37,615 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 15:00:37,616 INFO [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(3304): Received CLOSE for a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:37,615 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 15:00:37,615 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 15:00:37,615 INFO [RS:2;jenkins-hbase4:42505] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 15:00:37,616 DEBUG [RS:1;jenkins-hbase4:33165] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14a0e377 to 127.0.0.1:58940 2023-02-11 15:00:37,616 INFO [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:37,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a3f59341dd39a469cff752a69644f1c5, disabling compactions & flushes 2023-02-11 15:00:37,616 DEBUG [RS:0;jenkins-hbase4:41207] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62f1e446 to 127.0.0.1:58940 2023-02-11 15:00:37,616 DEBUG [RS:0;jenkins-hbase4:41207] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:37,616 INFO [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:37,616 INFO [RS:0;jenkins-hbase4:41207] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 15:00:37,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:37,616 DEBUG [RS:1;jenkins-hbase4:33165] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:37,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:37,617 INFO [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33165,1676127604743; all regions closed. 2023-02-11 15:00:37,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. after waiting 0 ms 2023-02-11 15:00:37,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:37,617 INFO [RS:0;jenkins-hbase4:41207] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 15:00:37,616 DEBUG [RS:2;jenkins-hbase4:42505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65397fd3 to 127.0.0.1:58940 2023-02-11 15:00:37,617 INFO [RS:0;jenkins-hbase4:41207] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 15:00:37,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a3f59341dd39a469cff752a69644f1c5 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-11 15:00:37,617 INFO [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-11 15:00:37,617 DEBUG [RS:2;jenkins-hbase4:42505] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:37,617 INFO [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42505,1676127604785; all regions closed. 2023-02-11 15:00:37,617 INFO [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-11 15:00:37,618 DEBUG [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1479): Online Regions={a3f59341dd39a469cff752a69644f1c5=hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5., 1588230740=hbase:meta,,1.1588230740} 2023-02-11 15:00:37,618 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 15:00:37,618 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 15:00:37,618 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 15:00:37,618 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 15:00:37,618 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 15:00:37,618 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-11 15:00:37,621 DEBUG [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1505): Waiting on 1588230740, a3f59341dd39a469cff752a69644f1c5 2023-02-11 15:00:37,628 DEBUG [RS:1;jenkins-hbase4:33165] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/oldWALs 2023-02-11 15:00:37,628 INFO [RS:1;jenkins-hbase4:33165] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33165%2C1676127604743:(num 1676127606305) 2023-02-11 15:00:37,628 DEBUG [RS:1;jenkins-hbase4:33165] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:37,628 DEBUG [RS:2;jenkins-hbase4:42505] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/oldWALs 2023-02-11 15:00:37,628 INFO [RS:1;jenkins-hbase4:33165] regionserver.LeaseManager(133): Closed leases 2023-02-11 15:00:37,628 INFO [RS:2;jenkins-hbase4:42505] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42505%2C1676127604785:(num 1676127606304) 2023-02-11 15:00:37,628 DEBUG [RS:2;jenkins-hbase4:42505] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:37,629 INFO [RS:2;jenkins-hbase4:42505] regionserver.LeaseManager(133): Closed leases 2023-02-11 15:00:37,629 INFO [RS:1;jenkins-hbase4:33165] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 15:00:37,629 INFO [RS:2;jenkins-hbase4:42505] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 15:00:37,629 INFO [RS:1;jenkins-hbase4:33165] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 15:00:37,630 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 15:00:37,630 INFO [RS:1;jenkins-hbase4:33165] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 15:00:37,630 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 15:00:37,630 INFO [RS:2;jenkins-hbase4:42505] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 15:00:37,630 INFO [RS:1;jenkins-hbase4:33165] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 15:00:37,630 INFO [RS:2;jenkins-hbase4:42505] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 15:00:37,630 INFO [RS:2;jenkins-hbase4:42505] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 15:00:37,631 INFO [RS:1;jenkins-hbase4:33165] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33165 2023-02-11 15:00:37,632 INFO [RS:2;jenkins-hbase4:42505] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42505 2023-02-11 15:00:37,641 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5/.tmp/info/4b4bd90bc4654b3eb8a8df4e3d1098a5 2023-02-11 15:00:37,641 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/.tmp/info/02deff5d89054a97ba37bc0b6e366c98 2023-02-11 15:00:37,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5/.tmp/info/4b4bd90bc4654b3eb8a8df4e3d1098a5 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5/info/4b4bd90bc4654b3eb8a8df4e3d1098a5 2023-02-11 15:00:37,644 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 02deff5d89054a97ba37bc0b6e366c98 2023-02-11 15:00:37,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5/info/4b4bd90bc4654b3eb8a8df4e3d1098a5, entries=2, sequenceid=6, filesize=4.8 K 2023-02-11 15:00:37,648 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for a3f59341dd39a469cff752a69644f1c5 in 31ms, sequenceid=6, compaction requested=false 2023-02-11 15:00:37,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/namespace/a3f59341dd39a469cff752a69644f1c5/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-11 15:00:37,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:37,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a3f59341dd39a469cff752a69644f1c5: Waiting for close lock at 1676127637616Running coprocessor pre-close hooks at 1676127637616Disabling compacts and flushes for region at 1676127637616Disabling writes for close at 1676127637617 (+1 ms)Obtaining lock to block concurrent updates at 1676127637617Preparing flush snapshotting stores in a3f59341dd39a469cff752a69644f1c5 at 1676127637617Finished memstore snapshotting hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676127637617Flushing stores of hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. at 1676127637619 (+2 ms)Flushing a3f59341dd39a469cff752a69644f1c5/info: creating writer at 1676127637619Flushing a3f59341dd39a469cff752a69644f1c5/info: appending metadata at 1676127637627 (+8 ms)Flushing a3f59341dd39a469cff752a69644f1c5/info: closing flushed file at 1676127637627Flushing a3f59341dd39a469cff752a69644f1c5/info: reopening flushed file at 1676127637645 (+18 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for a3f59341dd39a469cff752a69644f1c5 in 31ms, sequenceid=6, compaction requested=false at 1676127637648 (+3 ms)Writing region close event to WAL at 1676127637650 (+2 ms)Running coprocessor post-close hooks at 1676127637654 (+4 ms)Closed at 1676127637655 (+1 ms) 2023-02-11 15:00:37,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676127606723.a3f59341dd39a469cff752a69644f1c5. 2023-02-11 15:00:37,656 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/.tmp/rep_barrier/ad80c21cf3884dea90b2c4ea25b3fa18 2023-02-11 15:00:37,659 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ad80c21cf3884dea90b2c4ea25b3fa18 2023-02-11 15:00:37,671 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/.tmp/table/5b111f442dc744b4af54b298c002dbcd 2023-02-11 15:00:37,673 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b111f442dc744b4af54b298c002dbcd 2023-02-11 15:00:37,674 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/.tmp/info/02deff5d89054a97ba37bc0b6e366c98 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/info/02deff5d89054a97ba37bc0b6e366c98 2023-02-11 15:00:37,677 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 02deff5d89054a97ba37bc0b6e366c98 2023-02-11 15:00:37,677 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/info/02deff5d89054a97ba37bc0b6e366c98, entries=64, sequenceid=307, filesize=11.8 K 2023-02-11 15:00:37,678 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/.tmp/rep_barrier/ad80c21cf3884dea90b2c4ea25b3fa18 as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/rep_barrier/ad80c21cf3884dea90b2c4ea25b3fa18 2023-02-11 15:00:37,680 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 15:00:37,680 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 15:00:37,680 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 15:00:37,681 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ad80c21cf3884dea90b2c4ea25b3fa18 2023-02-11 15:00:37,681 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/rep_barrier/ad80c21cf3884dea90b2c4ea25b3fa18, entries=18, sequenceid=307, filesize=7.0 K 2023-02-11 15:00:37,681 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/.tmp/table/5b111f442dc744b4af54b298c002dbcd as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/table/5b111f442dc744b4af54b298c002dbcd 2023-02-11 15:00:37,684 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b111f442dc744b4af54b298c002dbcd 2023-02-11 15:00:37,684 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/table/5b111f442dc744b4af54b298c002dbcd, entries=22, sequenceid=307, filesize=7.0 K 2023-02-11 15:00:37,685 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 67ms, sequenceid=307, compaction requested=false 2023-02-11 15:00:37,692 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-11 15:00:37,692 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 15:00:37,694 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 15:00:37,694 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676127637618Running coprocessor pre-close hooks at 1676127637618Disabling compacts and flushes for region at 1676127637618Disabling writes for close at 1676127637618Obtaining lock to block concurrent updates at 1676127637618Preparing flush snapshotting stores in 1588230740 at 1676127637618Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141010, getHeapSize=217528, getOffHeapSize=0, getCellsCount=997 at 1676127637618Flushing stores of hbase:meta,,1.1588230740 at 1676127637619 (+1 ms)Flushing 1588230740/info: creating writer at 1676127637619Flushing 1588230740/info: appending metadata at 1676127637629 (+10 ms)Flushing 1588230740/info: closing flushed file at 1676127637629Flushing 1588230740/rep_barrier: creating writer at 1676127637645 (+16 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676127637647 (+2 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676127637647Flushing 1588230740/table: creating writer at 1676127637659 (+12 ms)Flushing 1588230740/table: appending metadata at 1676127637661 (+2 ms)Flushing 1588230740/table: closing flushed file at 1676127637661Flushing 1588230740/info: reopening flushed file at 1676127637674 (+13 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676127637678 (+4 ms)Flushing 1588230740/table: reopening flushed file at 1676127637682 (+4 ms)Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 67ms, sequenceid=307, compaction requested=false at 1676127637685 (+3 ms)Writing region close event to WAL at 1676127637690 (+5 ms)Running coprocessor post-close hooks at 1676127637692 (+2 ms)Closed at 1676127637694 (+2 ms) 2023-02-11 15:00:37,694 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-11 15:00:37,719 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:37,719 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 15:00:37,719 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 15:00:37,719 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:37,719 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42505,1676127604785 2023-02-11 15:00:37,720 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 15:00:37,720 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@66312f19 rejected from java.util.concurrent.ThreadPoolExecutor@620ea707[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:37,721 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 15:00:37,721 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:37,721 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@79c7c0c0 rejected from java.util.concurrent.ThreadPoolExecutor@620ea707[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:37,721 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@14d7fcb4 rejected from java.util.concurrent.ThreadPoolExecutor@10e86af6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:37,721 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:37,721 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33165,1676127604743 2023-02-11 15:00:37,721 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@140da52 rejected from java.util.concurrent.ThreadPoolExecutor@620ea707[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:37,721 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33165,1676127604743] 2023-02-11 15:00:37,722 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33165,1676127604743; numProcessing=1 2023-02-11 15:00:37,723 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33165,1676127604743 already deleted, retry=false 2023-02-11 15:00:37,723 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,33165,1676127604743 expired; onlineServers=2 2023-02-11 15:00:37,723 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42505,1676127604785] 2023-02-11 15:00:37,723 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42505,1676127604785; numProcessing=2 2023-02-11 15:00:37,724 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42505,1676127604785 already deleted, retry=false 2023-02-11 15:00:37,724 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42505,1676127604785 expired; onlineServers=1 2023-02-11 15:00:37,821 INFO [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41207,1676127604675; all regions closed. 2023-02-11 15:00:37,827 DEBUG [RS:0;jenkins-hbase4:41207] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/oldWALs 2023-02-11 15:00:37,827 INFO [RS:0;jenkins-hbase4:41207] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41207%2C1676127604675.meta:.meta(num 1676127606438) 2023-02-11 15:00:37,832 DEBUG [RS:0;jenkins-hbase4:41207] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/oldWALs 2023-02-11 15:00:37,832 INFO [RS:0;jenkins-hbase4:41207] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41207%2C1676127604675:(num 1676127606305) 2023-02-11 15:00:37,832 DEBUG [RS:0;jenkins-hbase4:41207] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:37,832 INFO [RS:0;jenkins-hbase4:41207] regionserver.LeaseManager(133): Closed leases 2023-02-11 15:00:37,832 INFO [RS:0;jenkins-hbase4:41207] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 15:00:37,832 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 15:00:37,833 INFO [RS:0;jenkins-hbase4:41207] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41207 2023-02-11 15:00:37,835 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41207,1676127604675 2023-02-11 15:00:37,835 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 15:00:37,836 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41207,1676127604675] 2023-02-11 15:00:37,836 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41207,1676127604675; numProcessing=3 2023-02-11 15:00:37,838 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41207,1676127604675 already deleted, retry=false 2023-02-11 15:00:37,838 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,41207,1676127604675 expired; onlineServers=0 2023-02-11 15:00:37,838 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38517,1676127603486' ***** 2023-02-11 15:00:37,838 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-11 15:00:37,839 DEBUG [M:0;jenkins-hbase4:38517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13897c3b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 15:00:37,839 INFO [M:0;jenkins-hbase4:38517] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38517,1676127603486 2023-02-11 15:00:37,839 INFO [M:0;jenkins-hbase4:38517] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38517,1676127603486; all regions closed. 2023-02-11 15:00:37,839 DEBUG [M:0;jenkins-hbase4:38517] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 15:00:37,839 DEBUG [M:0;jenkins-hbase4:38517] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-11 15:00:37,839 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-11 15:00:37,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676127605909] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676127605909,5,FailOnTimeoutGroup] 2023-02-11 15:00:37,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676127605911] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676127605911,5,FailOnTimeoutGroup] 2023-02-11 15:00:37,839 DEBUG [M:0;jenkins-hbase4:38517] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-11 15:00:37,840 INFO [M:0;jenkins-hbase4:38517] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-11 15:00:37,840 INFO [M:0;jenkins-hbase4:38517] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-11 15:00:37,840 INFO [M:0;jenkins-hbase4:38517] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-11 15:00:37,840 DEBUG [M:0;jenkins-hbase4:38517] master.HMaster(1502): Stopping service threads 2023-02-11 15:00:37,840 INFO [M:0;jenkins-hbase4:38517] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-11 15:00:37,841 ERROR [M:0;jenkins-hbase4:38517] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-11 15:00:37,841 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 15:00:37,841 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 15:00:37,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 15:00:37,841 INFO [M:0;jenkins-hbase4:38517] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-11 15:00:37,841 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-11 15:00:37,842 DEBUG [M:0;jenkins-hbase4:38517] zookeeper.ZKUtil(398): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-11 15:00:37,842 WARN [M:0;jenkins-hbase4:38517] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-11 15:00:37,842 INFO [M:0;jenkins-hbase4:38517] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-11 15:00:37,842 INFO [M:0;jenkins-hbase4:38517] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-11 15:00:37,842 DEBUG [M:0;jenkins-hbase4:38517] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 15:00:37,842 INFO [M:0;jenkins-hbase4:38517] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 15:00:37,842 DEBUG [M:0;jenkins-hbase4:38517] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 15:00:37,842 DEBUG [M:0;jenkins-hbase4:38517] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 15:00:37,842 DEBUG [M:0;jenkins-hbase4:38517] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 15:00:37,842 INFO [M:0;jenkins-hbase4:38517] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=845.59 KB heapSize=1.01 MB 2023-02-11 15:00:37,856 INFO [M:0;jenkins-hbase4:38517] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=845.59 KB at sequenceid=2120 (bloomFilter=true), to=hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/22bd8027b19343b0abb4015a42f21eac 2023-02-11 15:00:37,858 INFO [M:0;jenkins-hbase4:38517] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 22bd8027b19343b0abb4015a42f21eac 2023-02-11 15:00:37,859 DEBUG [M:0;jenkins-hbase4:38517] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/22bd8027b19343b0abb4015a42f21eac as hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/22bd8027b19343b0abb4015a42f21eac 2023-02-11 15:00:37,861 INFO [M:0;jenkins-hbase4:38517] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 22bd8027b19343b0abb4015a42f21eac 2023-02-11 15:00:37,861 INFO [M:0;jenkins-hbase4:38517] regionserver.HStore(1080): Added hdfs://localhost:37623/user/jenkins/test-data/4706581b-ef31-0dbf-a522-801077a83a92/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/22bd8027b19343b0abb4015a42f21eac, entries=282, sequenceid=2120, filesize=30.8 K 2023-02-11 15:00:37,862 INFO [M:0;jenkins-hbase4:38517] regionserver.HRegion(2947): Finished flush of dataSize ~845.59 KB/865888, heapSize ~1.01 MB/1062696, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=2120, compaction requested=false 2023-02-11 15:00:37,863 INFO [M:0;jenkins-hbase4:38517] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 15:00:37,863 DEBUG [M:0;jenkins-hbase4:38517] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676127637842Disabling compacts and flushes for region at 1676127637842Disabling writes for close at 1676127637842Obtaining lock to block concurrent updates at 1676127637842Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676127637842Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=865888, getHeapSize=1062696, getOffHeapSize=0, getCellsCount=2607 at 1676127637843 (+1 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676127637843Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676127637844 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676127637848 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676127637848Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676127637859 (+11 ms)Finished flush of dataSize ~845.59 KB/865888, heapSize ~1.01 MB/1062696, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=2120, compaction requested=false at 1676127637862 (+3 ms)Writing region close event to WAL at 1676127637863 (+1 ms)Closed at 1676127637863 2023-02-11 15:00:37,866 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 15:00:37,866 INFO [M:0;jenkins-hbase4:38517] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-11 15:00:37,867 INFO [M:0;jenkins-hbase4:38517] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38517 2023-02-11 15:00:37,868 DEBUG [M:0;jenkins-hbase4:38517] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,38517,1676127603486 already deleted, retry=false 2023-02-11 15:00:38,012 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 15:00:38,012 INFO [M:0;jenkins-hbase4:38517] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38517,1676127603486; zookeeper connection closed. 2023-02-11 15:00:38,012 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f604df8 rejected from java.util.concurrent.ThreadPoolExecutor@400725af[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:38,012 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): master:38517-0x10043e5717e0000, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 15:00:38,012 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1f620c9c rejected from java.util.concurrent.ThreadPoolExecutor@400725af[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:38,112 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 15:00:38,112 INFO [RS:0;jenkins-hbase4:41207] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41207,1676127604675; zookeeper connection closed. 2023-02-11 15:00:38,112 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@39c58faf rejected from java.util.concurrent.ThreadPoolExecutor@3dfa0c4f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:38,112 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:41207-0x10043e5717e0001, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 15:00:38,112 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5c672e2b rejected from java.util.concurrent.ThreadPoolExecutor@3dfa0c4f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:38,112 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@79537f67] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@79537f67 2023-02-11 15:00:38,212 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 15:00:38,212 INFO [RS:1;jenkins-hbase4:33165] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33165,1676127604743; zookeeper connection closed. 2023-02-11 15:00:38,212 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f6cc185 rejected from java.util.concurrent.ThreadPoolExecutor@10e86af6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:38,212 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:33165-0x10043e5717e0002, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 15:00:38,212 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3719cf22 rejected from java.util.concurrent.ThreadPoolExecutor@10e86af6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:38,213 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6eb76138] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6eb76138 2023-02-11 15:00:38,312 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 15:00:38,312 INFO [RS:2;jenkins-hbase4:42505] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42505,1676127604785; zookeeper connection closed. 2023-02-11 15:00:38,312 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@a5c389 rejected from java.util.concurrent.ThreadPoolExecutor@620ea707[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:38,313 DEBUG [Listener at localhost/39605-EventThread] zookeeper.ZKWatcher(600): regionserver:42505-0x10043e5717e0003, quorum=127.0.0.1:58940, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 15:00:38,313 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4bffb3b2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4bffb3b2 2023-02-11 15:00:38,313 ERROR [Listener at localhost/39605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1a443078 rejected from java.util.concurrent.ThreadPoolExecutor@620ea707[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 15:00:38,313 INFO [Listener at localhost/39605] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-11 15:00:38,313 WARN [Listener at localhost/39605] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 15:00:38,316 INFO [Listener at localhost/39605] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 15:00:38,421 WARN [BP-1814316695-172.31.14.131-1676127600293 heartbeating to localhost/127.0.0.1:37623] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 15:00:38,422 WARN [BP-1814316695-172.31.14.131-1676127600293 heartbeating to localhost/127.0.0.1:37623] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1814316695-172.31.14.131-1676127600293 (Datanode Uuid bf0add4a-0a9f-4550-8ba1-632114d3a2f1) service to localhost/127.0.0.1:37623 2023-02-11 15:00:38,423 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data5/current/BP-1814316695-172.31.14.131-1676127600293] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 15:00:38,424 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data6/current/BP-1814316695-172.31.14.131-1676127600293] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 15:00:38,425 WARN [Listener at localhost/39605] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 15:00:38,427 INFO [Listener at localhost/39605] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 15:00:38,531 WARN [BP-1814316695-172.31.14.131-1676127600293 heartbeating to localhost/127.0.0.1:37623] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 15:00:38,531 WARN [BP-1814316695-172.31.14.131-1676127600293 heartbeating to localhost/127.0.0.1:37623] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1814316695-172.31.14.131-1676127600293 (Datanode Uuid 2ed47ade-61b2-412c-b2fa-38a8167f2f53) service to localhost/127.0.0.1:37623 2023-02-11 15:00:38,532 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data3/current/BP-1814316695-172.31.14.131-1676127600293] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 15:00:38,532 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data4/current/BP-1814316695-172.31.14.131-1676127600293] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 15:00:38,533 WARN [Listener at localhost/39605] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 15:00:38,535 INFO [Listener at localhost/39605] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 15:00:38,639 WARN [BP-1814316695-172.31.14.131-1676127600293 heartbeating to localhost/127.0.0.1:37623] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 15:00:38,639 WARN [BP-1814316695-172.31.14.131-1676127600293 heartbeating to localhost/127.0.0.1:37623] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1814316695-172.31.14.131-1676127600293 (Datanode Uuid ffa7b8bb-5ab6-498c-b50f-1688dc4647c1) service to localhost/127.0.0.1:37623 2023-02-11 15:00:38,640 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data1/current/BP-1814316695-172.31.14.131-1676127600293] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 15:00:38,640 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4ec29e20-5454-c6bc-54e5-fd044f4f4aab/cluster_1c63cfe1-024f-dc07-f0c3-e6244c5c236f/dfs/data/data2/current/BP-1814316695-172.31.14.131-1676127600293] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 15:00:38,667 INFO [Listener at localhost/39605] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 15:00:38,794 INFO [Listener at localhost/39605] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-11 15:00:38,867 INFO [Listener at localhost/39605] hbase.HBaseTestingUtility(1293): Minicluster is down